Getting Structured Data from Web Pages with AI and Re-usable Specs

As I say from time to time, every now and then I build something as a proof of concept and the result is... ok? I'm not really sure if the idea actually made sense and if the final result is usable, but the exploration was interesting enough that I think it warrants sharing with you good people. That's what today's post is. This demo all began with a simple idea - how do you handle getting RSS like content from a site that doesn't actually have an RSS feed? This typically relies on looking at the page as data to be parsed. For a while now, there have been multiple good ways to turn a web page into structured data. As an example, AgentQL lets use a GraphQL-like syntax to parse web pages. I blogged about it here: Extracting Data from Web Pages with AgentQL and BoxLang. Another way of handling this is via an API from Diffbot - the List API. This is focused on "list" type pages, so not so much a recipe but a page with a set of content, like a news homepage for example. The Diffbot team…

Commenting on Bubbles has been removed. Why? If you want to talk about this post, write to the author, use the comments on their blog if it has them, or reply with a "RE:" post on your own blog. Suggestions? feedback@bubbles.town