Read entire HTML source using Nokogiri in Ruby

How would I obtain the full source of a webpage so that I can access the price? I'm trying to get the CSS class ".old-price" from the link "https://www.udemy.com/video-editing-for-instructors/?ccManual=&dtcode=RI7hLVu42dR7&couponCode=9dollarenrollment?

I think the reason I'm having problems is due to this Udemy code

   data-wrapcss = "static-content-wrapper payment-popup"
   data-passDtCode="true"
   data-enableLoader="true"
   data-purpose="take-this-course-button"
           data-overlayClosable="false"

And Nokogiri is skipping the code in that container which happenes to have some info I want. Here's my code trying to make it work:

require 'nokogiri'
require 'open-uri' 
page = Nokogiri::HTML(open("https://www.udemy.com/video-editing-for-instructors/?ccManual=&dtcode=RI7hLVu42dR7&couponCode=9dollarenrollment"))
puts page.css('.old-price') #this doesn't showcase anything, I want it to show the price of $89

Solved

If you download the source code with a command-line tool like wget or curl you will find that there is no such thing as old-price in the source code. So the class and that old price likely appears by the magic of the browser interpreting JavaScript. Nokogiri does not interpret JavaScript so it won't find the class you are looking for.

If you want to automate scraping of JavaScript-enabled web pages you have to use a tool that understands JavaScript. The test framework Capybara can probably be used for this together with a JavaScript-aware driver like WebKit or Poltergeist. I am sure there are other alternatives out there also.

Once you get the markup you can still use Nokogiri if you want. It is a nice parser. Capybara has some nice finders built on top of Nokogiri.

Alternative approach: if you start your browser's developer tools and go to the network tab you will see that there is a huge amount of requests from the web server to www.udemy.com. One of them will contain that old price either as markup or as data like JSON or XML. This may be an easier approach. Or not - the API may be secure so it is not possible to call it from the command line.

The element would be:

page.at('.old-price')

The HTML (after libxml has it's way with it) would be:

page.to_s

and the original HTML would be:

open(url).read