I use the built-in IE control to programmatically navigate webpages and scrape for info.  Richard probably has lessons on the IE control, but I don't remember where.  Below is a highly simplified summary of what this all entails.

First, you need to look at the webpage source (by viewing the page source on a web browser) and study the tag structures, which may be nested, sometimes several levels deep.

Then you add the IE control to Access by going to the VBA editor -> Tools -> References, and add "Microsoft Internet Controls."

Then you write code that looks like below, which declare the variables, set up the browser object, navigate to the specific tags (which is the hardest part), then glean the actual info from the tags.  This code scrapes the plot summary of a movie.

Details


    ' Declare variables.
    Dim browser As InternetExplorer, doc As HTMLDocument, v As String
    
    ' Set up IE control; keep loop this until success.
    On Error Resume Next
    Do While (browser Is Nothing)
        Set browser = New InternetExplorer      
    Loop
    On Error GoTo 0
    browser.Visible = False
    
    ' Navigate to IMDB movie page.
    browser.Navigate "https://www.imdb.com/title/tt0050083/reference/"
    Do While browser.Busy: Loop
    
    ' Scrape text from plot summary.
    v = ""
    Set objs = doc.getElementsByClassName("titlereference-section-overview")
    v = objs.item(0).getElementsByTagName("div").item(0).innerText


Below is a video of my Access movie database that uses web-scraping to get info from IMDb.  First it scrapes the basic movie info (title, year, genre, countries of origin, etc.), then it scrapes the cast and crew.

     https://www.youtube.com/watch?v=XF0KFen8hwI

My code above references "titlereference-section-overview".  That is the name of the tag on the webpage source.  The plot summary is within this tag (see picture below).  When the code runs, the variable v will have the plot summary text: "The jury in a New York City murder trial..."

Many developers avoid web-scraping because if the HTML tag structures change (as they often do, since websites are regularly re-designed); and when that happens, you have to re-code everything.  This is not something you want to rely on for a mission-critical application.  I only do it because my movie database is only a hobby.

If they have APIs available to them, developers would prefer API for retrieving web info to web-scraping.

IMDb has a setting for members to always show webpages in the old, standard format.  This is mainly intended as a style choice for users, but it indirectly helps web-scrapers -- because the standard format means its tag structures will likely never change (or not so frequently).  This is one thing to consider if you ever resort to web-scraping.

I'm still writing in the "highly simplified summary" mode despite my apparent verbosity.  All this info is just a tip of the iceberg.  As my video shows, the code could get really long and complicated.

Kevin, this is a lot of info, I sure appreciate you taking the time and effort to put it together. I am not using this for critical data, I'm using it right now for a database that I started on the techHelp videos. So far I have it down to three clicks to get 90% of the data and then have to look at the webpage to get the reference numbers for the Quiz and the Transcript info. Trying to get it to either one or two clicks or if possible totally auto.

Thanks Kevin

If memory serves the Access Web Sync Seminar covers some techniques.

(Previous post for ref: Web scrapping a link)