Scraping webpage with a table rendered using javascript utilizing Selenium Webdriver

Question

the source html of the page i am trying to scrape

Iam trying to scrape a webtable that is rendered using certain javascripts using Selenium Webdriver

driver.get("http://xxxxx:xxxxxxxx@xxxxxx-
xxxxxx.grid.xxxxxx.com/Windchill/app/#ptc1/comp/queue.table");
driver.manage().timeouts().implicitlyWait(20, TimeUnit.SECONDS);
List<WebElement> k=driver.findElements(By.xpath("//*[@id='queue.table']"));
System.out.println(k.size());
System.out.println(k.get(0).getText());

k.size() returns 1 and when i run get text it returns only some entries from the table

Actual table and entries the total rows are 135

after running i get as follows

              Queue Management
 Loading...

 Name
 Type
 Status
 Enabled
 Group
 Total Entries
 Waiting Entries
 Severe/Failed Entries
 DeleteCompletedWorkItemsQueu e
 Process
 Started
 Enabled
 Default
 0
 0
 0
 DeliveryStatusOnStartup
 Process
 Started
 Enabled
 Default
 0
 0
 0
 DTODeliverablesQueue
 Process
 Started
 Enabled
 Default
 0
 0
 0
 DTOOffPeakQueue
 Process
 Started
 Enabled
 Default
 0
 0
 0
Loading.........

I get 25 entries of the table and rest is not present I am unable to understand why am i getting "Loading....."

That just means that data is not there. Some sites only load data when you scrolldown. So either you need to scroll down and then do it or you need to use wait if the page is still loading — Tarun Lalwani
– Tarun Lalwani, Commented Sep 21, 2017 at 10:09
is table is loading when scroll down or without scroll down? — Murthi
– Murthi, Commented Sep 21, 2017 at 11:09
I am trying to fetch the data using selenium so i am unaware as to how to scroll down programmatically — joseph francis
– joseph francis, Commented Sep 21, 2017 at 11:15

undetected Selenium · Accepted Answer · 2017-09-21 12:46:50Z

1

I think by using List<WebElement> k=driver.findElements(By.xpath("//*[@id='queue.table']")); we are trying to make out a list with too many unwanted items in the list. Rather, I feel it would be effective to get hold of the nodes within <td> tags which contains the indented values and save into the list. Next we can iterate over the list and use either getText() method or getAttribute() method to retrieve the text as follows:

driver.get("http://xxxxx:[email protected]/Windchill/app/#ptc1/comp/queue.table");
driver.manage().timeouts().implicitlyWait(20, TimeUnit.SECONDS);
List<WebElement> k = driver.findElements(By.xpath("//*[@id='queue.table']//tr"));
System.out.println(k.size());
for (WebElement my_element:k)
    {
        String innerhtml = my_element.getAttribute("innerHTML");
        System.out.println("Value from Table is : "+innerhtml); 
    }

edited Sep 21, 2017 at 12:46

answered Sep 21, 2017 at 11:56

undetected Selenium

194k44 gold badges304 silver badges387 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

joseph francis Over a year ago

I was looking for the text int the table rows and modifying to my_element.getText was also not that helpful as i get even less values in a repeated fashion

undetected Selenium Over a year ago

@josephfrancis As you can see, I have changed my code and now picking up the <tr> nodes and retrieving the text not through getText() but through getAttribute("innerHTML"). Please Accept the Answer if it caters to your Question.

joseph francis Over a year ago

Unfortunately no;appreciate your effort

Collectives™ on Stack Overflow

Scraping webpage with a table rendered using javascript utilizing Selenium Webdriver

1 Answer 1

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related