Chrome Driver Headless Option Is Not Working For Link?

--headless option is not working for some link I don't know why but working for other links. Need of headless is I have to run this crawler on AWS instance which must required head

Solution 1:

As the latest version of chromedriver(After ChromeDriver 79.0.3945.16) fixed the issue element not clickable on headless mode.

Download the latest version of chromedriver.Hopefully your problem will be solved.

Here is the Changelog

• Fixed ChromeDriver crash caused by javascript alert fired during command execution

• Fixed a bug causing Chromedriver to lock when an alert is fired while taking a screenshot

• Removed --ignore-certificate-errors from Chrome launch command Changed platform and platformName to windows on Win10

• Fixed undefined window.navigator.webdriver when "enable-automation" is excluded

• Fixed WPT test "test_not_editable_inputs[hidden]"

• Fixed "Element is not clickable" when using headless mode

Chromedriver changelog and download linkhttps://chromedriver.chromium.org/downloads

EDITED For aws You need to follow these steps

First, you need to install chrome using the following steps

sudo curl -sS -o - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add
sudo echo "deb [arch=amd64]  http://dl.google.com/linux/chrome/deb/ stable main" >>/etc/apt/sources.list.d/google-chrome.list
sudo apt-get-y update
sudo apt-get-y install google-chrome-stable

Then Download chromedriver

wget https://chromedriver.storage.googleapis.com/index.html?path=79.0.3945.16/
unzip chromedriver_linux64.zip

Now add permission and move binary file

sudo mv chromedriver /usr/bin/chromedriver
sudo chown root:root /usr/bin/chromedriver
sudo chmod +x /usr/bin/chromedriver

For opening chrome change the binary path options.binary_location and add argument options.add_argument('--headless')

NB: Do not forget to install java

Solution 2:

After searching a lot i notice those web-sites which send JS code first they have problem in opening in headless argument so i come up with solution to use virtualDisplay ..

Run this command in terminal or windows cmd or power-shell to install this

apt-get -y install xvfb    
pip install PyVirtualDisplay
pip install xvfbwrapper

and this is the code you need to put in your code

from pyvirtualdisplay importDisplaydisplay= Display(visible=0, size=(800, 600))
display.start()

after you code in the end

display.stop()