This topic was automatically generated from Slack. You can find the original thread here.
Hello sophisticated developers of the world!
I am a realestate developer based in Australia and I just discovered Pipedream tonight while watching random tech & AI videos on Youtube.
I wondered if Pipedream could be used to “scan” a local realestate website called https://www.realestate.com.au/ here and report data into Excel based on certain search parameters.
For example,
“Search for suburbs where the price of house or land >800m2 is being sold for <$1m and new townhouses are being sold for >$1.2m”
I know absolutely nothing about coding or really any of this stuff for that matter. I am a complete beginner in this space.
Is something like this possible with Pipedream, and if so, what is the best way for me to go about having it created?
Sergio here!! Sophisticated developer from Mexico. I collaborate with Pipedream by onboarding apps to the Pipedream marketplace.
I can confirm you that what you want to do is perfectly possible. I can imagine two approaches here.
One is that the real state website you use has an API available (which stands for Application Programming Interface… Think of an User Interface, but tou do the interfacing with code!) On Pipedream, you can connect anything that exposes a public API (meaning is open for the general public, not closed to the apps developers of said website or service). Second approach is that you scan for the data using an “scrapping strategy” via code. Scrapping is the act of parsing, extracting data programmatically without an API (meaning you need to “navigate” for the data programmatically, sometimes using a so called headless browser, meaning a browser that doesn’t have a UI and it’s,driven, controlled via code).
Either way you go, you’ll need to save the data into excel. Excel online has a an API, which it’s self is part of Microsoft Graph APIs a family of APIs accessing Microsoft products. Here are some scrapping you could start leveraging, and a link to Excel app integration at Pipedream
You can request realestate.com.au here https://github.com/PipedreamHQ/pipedream/issues/new/choose (or any other app integration for that matter) I’ll check on it later though (having breakfast right now) btw, isn’t too late in Australia aka the west Island (or so says a kiwi friend I met like two years ago)
Hi Sergio,
Thank you I really appreciate your detailed response. I will have a look at some of the options you’ve mentioned here. Might take me a bit of playing around to understand how it all works and to put something together but I’m sure I’ll get there in the end. Thank you!
It turns out I’ve looked out for an Realstate.com.au app integration at Pipedream and apparently there is no official API from Realstate.com.au directly.
But there is out there a developer company named API Dojo, it’s a software company that “just” develops APIs. They offer the Realstate API at RapidAPI. RapidAPI is itself a martketplace where developers sell their APIs as a service *you pay certain amount of money and they credit certain amount of API calls). This is very normal as there is a whole API economy right now in the SaaS business.
Then you can head to Pipedream and connect your RapidAPI key via the RapidAPI app> https://pipedream.com/apps/rapidapi you-ll have to adjust for your realstate case.
Rapid API offers a free tier of 500 APi calls / month, so is good for you to start playing around.
Hi Sergio,
Wow I really appreciate you looking into this for me. Thank you!
Is there a way I can arrange assistance from an engineer with this to help me set it up?
This is my first time doing something like this and I’m a bit out of my depth on the technical aspects. In the mean time I’ll continue watching YouTube tutorials etc but thought I’d ask anyway.
Furthermore, with the site scrape request, I’m assuming once it’s set up it will look for certain fields of information and record that back into google sheets and then a further step would need to be taken within Google Sheets to set up a formula based on the conditions I’m looking for. For example, “suburbs with land <$1m AND townhouses selling for >$1.2m”
Or would with all be able to be collected and organised through pipedream without further data analysis in Google Sheets?
Regarding having some prebuild components that you could reuse (ex. Using the “realty in Au” API from ApiDojo) these could be done by the Pipedream components team. There’s a backlog on apps being creating components for.
Finally, you have your ideas right. It is common to copy data from a source data such as a website and sink the data to an staging database from which you query the data. Now, when it comes to Pipedream you could potentially use a feature called Data Store as the sink, and then query from there. Just be aware of limitations. In my experience though, I’ve done scrapping and saving to a third app specifically Excel, and Ms Sql Server. At the end it’ll greatly depend on the data size and the velocity you’d like when pulling the data for i.e. reporting, or visualization, charts and so on.
btw, isn’t too late in Australia aka the west Island (or so says a kiwi friend I met like two years ago)