Brian Lough | Scraping Data with an ESP8266 or an ESP32 @BrianLough | Uploaded January 2020 | Updated October 2024, 6 days ago.
Have you ever wanted to get data for your project, but there is no public API for it or cases like the Instagram API where the setup process for the is not very convenient? Well in this video we are going to look at 2 different options for scraping data from a website for your ESP8266 or ESP32 projects.
We'll look at Non-Public APIs and scraping the data directly.
---------------------------------
Info Links:
---------------------------------
Arduino sample request: github.com/witnessmenow/arduino-sample-api-request
Previous Video on API: youtube.com/watch?v=HUjFMVOpXBM
Team Trees code: github.com/witnessmenow/TeamTreesArduino
Crowdsupply NodeJS scraper: github.com/witnessmenow/crowdsupply-tp-scraper
---------------------------------
My Discord:
---------------------------------
Come hang out and talk about this video or any project you are working on!
discord.gg/nnezpvq
---------------------------------
Github Sponsorship:
---------------------------------
I have set up a Github Sponsorship page, it's similar style thing to patreon, but Github are offering to match all sponsorships for the first year! (you sponsor $2, I receive $4)
github.com/sponsors/witnessmenow
Some people have said they would prefer a once off way of supporting the channel, here is a Paypal link if you would prefer that!
paypal.me/bloughyt
---------------------------------
My Tindie Store:
---------------------------------
I've started to sell stuff on Tindie, check it out!
tindie.com/stores/brianlough
---------------------------------
Other Platforms:
---------------------------------
Twitter: @witnessmenow - twitter.com/witnessmenow
Instagram - instagram.com/brian_lough
Instructables: witnessmenow - instructables.com/member/witnessmenow
---------------------------------
Support the Channel:
---------------------------------
If you would like to support the channel, here are some ways you can.
1 - Like and comment on the videos - I really like hearing from you and apparently it helps!
2 - Share my videos - If you know somewhere that might be interested in my channel, please share it!
3 - Affiliate links - Where I get a small percentage of the purchase price but doesn't cost you anything extra.
Amazon.co.uk: amzn.to/2g7mJIO
Amazon.com : amzn.to/2ykDQ42
Amazon.de: amzn.to/2WMq7vF
Aliexpress: http://s.click.aliexpress.com/e/wdhojba
LCSC (where I buy most of my parts for Tindie stuff): lcsc.com/?promote=ODM5ZTB5NlkwNUN3bkNCZUtGb0ZlSWFKTGJIaTl3WThNWldwU0R6QndkZ3prNWdPemlTcG5XSi94U1Zpc0g5Y3MwYjFOUTNIZjJLbGhwWllsZ3N5QnR2WkZRdXMvdGVFcVhJWlJOdnNBV1U0RDZuY1M4TFlqZWNQYUo4RkN6VVJ2c0J4MTcvU3NEdw==
4 - My Amazon "store" - Really it's just a collection of stuff I use that I think are useful
amazon.co.uk/shop/brianlough
5 - Check out my Tindie Store
tindie.com/stores/brianlough
---------------------------------
My Tools & Kit*:
---------------------------------
Ender 3 Pro 3d Printer - I got one of these recently and I love it!
- UK : amzn.to/31bebXT
- US : amzn.to/2IlSLid
Ts-100 Soldering Iron - I have this a couple of years and I love it, I thinking about getting a second!
- UK : amzn.to/2nlOF23
- US : amzn.to/2jDcyxl
Logitech C920 HD 1080p Pro webcam - I use one of these for my videos
- UK: amzn.to/2pnxk9Z
- US: amzn.to/2BsLMiF
Blue Snowball USB Microphone
- UK: amzn.to/2C0x4TH
- US: amzn.to/2DEA85J
Self Adjusting Wire Strippers
- UK: amzn.to/2C0urkR
- US: amzn.to/2BXQAh1
*= Affiliate Links
Have you ever wanted to get data for your project, but there is no public API for it or cases like the Instagram API where the setup process for the is not very convenient? Well in this video we are going to look at 2 different options for scraping data from a website for your ESP8266 or ESP32 projects.
We'll look at Non-Public APIs and scraping the data directly.
---------------------------------
Info Links:
---------------------------------
Arduino sample request: github.com/witnessmenow/arduino-sample-api-request
Previous Video on API: youtube.com/watch?v=HUjFMVOpXBM
Team Trees code: github.com/witnessmenow/TeamTreesArduino
Crowdsupply NodeJS scraper: github.com/witnessmenow/crowdsupply-tp-scraper
---------------------------------
My Discord:
---------------------------------
Come hang out and talk about this video or any project you are working on!
discord.gg/nnezpvq
---------------------------------
Github Sponsorship:
---------------------------------
I have set up a Github Sponsorship page, it's similar style thing to patreon, but Github are offering to match all sponsorships for the first year! (you sponsor $2, I receive $4)
github.com/sponsors/witnessmenow
Some people have said they would prefer a once off way of supporting the channel, here is a Paypal link if you would prefer that!
paypal.me/bloughyt
---------------------------------
My Tindie Store:
---------------------------------
I've started to sell stuff on Tindie, check it out!
tindie.com/stores/brianlough
---------------------------------
Other Platforms:
---------------------------------
Twitter: @witnessmenow - twitter.com/witnessmenow
Instagram - instagram.com/brian_lough
Instructables: witnessmenow - instructables.com/member/witnessmenow
---------------------------------
Support the Channel:
---------------------------------
If you would like to support the channel, here are some ways you can.
1 - Like and comment on the videos - I really like hearing from you and apparently it helps!
2 - Share my videos - If you know somewhere that might be interested in my channel, please share it!
3 - Affiliate links - Where I get a small percentage of the purchase price but doesn't cost you anything extra.
Amazon.co.uk: amzn.to/2g7mJIO
Amazon.com : amzn.to/2ykDQ42
Amazon.de: amzn.to/2WMq7vF
Aliexpress: http://s.click.aliexpress.com/e/wdhojba
LCSC (where I buy most of my parts for Tindie stuff): lcsc.com/?promote=ODM5ZTB5NlkwNUN3bkNCZUtGb0ZlSWFKTGJIaTl3WThNWldwU0R6QndkZ3prNWdPemlTcG5XSi94U1Zpc0g5Y3MwYjFOUTNIZjJLbGhwWllsZ3N5QnR2WkZRdXMvdGVFcVhJWlJOdnNBV1U0RDZuY1M4TFlqZWNQYUo4RkN6VVJ2c0J4MTcvU3NEdw==
4 - My Amazon "store" - Really it's just a collection of stuff I use that I think are useful
amazon.co.uk/shop/brianlough
5 - Check out my Tindie Store
tindie.com/stores/brianlough
---------------------------------
My Tools & Kit*:
---------------------------------
Ender 3 Pro 3d Printer - I got one of these recently and I love it!
- UK : amzn.to/31bebXT
- US : amzn.to/2IlSLid
Ts-100 Soldering Iron - I have this a couple of years and I love it, I thinking about getting a second!
- UK : amzn.to/2nlOF23
- US : amzn.to/2jDcyxl
Logitech C920 HD 1080p Pro webcam - I use one of these for my videos
- UK: amzn.to/2pnxk9Z
- US: amzn.to/2BsLMiF
Blue Snowball USB Microphone
- UK: amzn.to/2C0x4TH
- US: amzn.to/2DEA85J
Self Adjusting Wire Strippers
- UK: amzn.to/2C0urkR
- US: amzn.to/2BXQAh1
*= Affiliate Links
![[TMT] - Using The Reset Button Of An ESP8266 As An Input](https://i.ytimg.com/vi/l9Gl1yKvMNg/hqdefault.jpg "[TMT] - Using The Reset Button Of An ESP8266 As An Input
I think this a great solution to how to force config mode when using the WiFiManager library.
Apologies, Im a little over the two minute mark here, but I was pretty happy with what I had covered so I thought an extra few seconds would be worth not cutting anything!
Also I upgraded to windows 10 recently and Im just noticing that I have not increased the font size in the Arduino IDE so its a little small.
Dobule Reset Detector library by datacute
https://github.com/datacute/DoubleResetDetector/
Example from YouTubeApi library:
https://github.com/witnessmenow/arduino-youtube-api/blob/master/examples/ESP8266/ChannelStatisticsWithWifiManagerAndDoubleReset/ChannelStatisticsWithWifiManagerAndDoubleReset.ino
Esp8266 Modules (affiliate links):
Wemos D1 Mini (Clone - the one in the video): http://s.click.aliexpress.com/e/uzFUnIe
NodeMCU (doesnt require soldering): http://s.click.aliexpress.com/e/AAyvJiA")
![Reducing latency between two ESP8266s for a blimp project! [Was live on Twitch]](https://i.ytimg.com/vi/p01qalPt3Js/hqdefault.jpg "Reducing latency between two ESP8266s for a blimp project! [Was live on Twitch]
I met Sophi Kravits at Dublin Hackaday Unconference and she was saying she was having some issues with latency on the her ESP8266 blimp project and I said we could take a look at what improvements we could make.
More info on Sophis project here:
https://hackaday.io/project/25509-remote-control-mini-blimp
Other Platforms:
Twitter: @witnessmenow - https://twitter.com/witnessmenow
Twtich: BrianLough - https://twitch.tv/brianlough
Instructables: witnessmenow - https://www.instructables.com/member/witnessmenow/
My Tools & Kit*:
Ts-100 Soldering Iron
- UK : http://amzn.to/2nlOF23
- US : http://amzn.to/2jDcyxl
Logitech C920 HD 1080p Pro webcam
- UK: http://amzn.to/2pnxk9Z
- US: http://amzn.to/2BsLMiF
Blue Snowball USB Microphone
- UK: http://amzn.to/2C0x4TH
- US: http://amzn.to/2DEA85J
Self Adjusting Wire Strippers
- UK: http://amzn.to/2C0urkR
- US: http://amzn.to/2BXQAh1
Support what I do by using my Amazon Affiliate links:
Amazon.co.uk: http://amzn.to/2g7mJIO
Amazon.com : http://amzn.to/2ykDQ42
*= Affiliate Links")
![Why I use the Portable Arduino IDE [Two Minute Tidbit]](https://i.ytimg.com/vi/sTTY_w7Cuzk/hqdefault.jpg "Why I use the Portable Arduino IDE [Two Minute Tidbit]
Two Minute Tidbits are roughly two minute long videos that try show you something interesting about working with Arduino.
In this video I’m going to show you the Portable version of the Arduino IDE and explain why it’s actually really useful, I even exclusively use the portable version where I can. It is really useful if you have library conflicts between different sketches
You basically just download a zipped version of the IDE and create a portable folder, but what I think is more interesting is why you might want to use it.
Links:
Arduino Portable Page: https://www.arduino.cc/en/Guide/PortableIDE
Arduino Download Page: https://www.arduino.cc/en/main/software
Potential workaround for Macs (I have not tested this!): https://forum.arduino.cc/index.php?topic=539551.msg3678998#msg3678998
My Discord:
Come hang out and talk about this video or any project you are working on!
https://discord.gg/nnezpvq
Github Sponsorship:
I have set up a Github Sponsorship page, its similar style thing to patreon, but Github are offering to match all sponsorships for the first year! (you sponsor $2, I receive $4)
https://github.com/sponsors/witnessmenow/
Some people have said they would prefer a once off way of supporting the channel, here is a Paypal link if you would prefer that!
https://www.paypal.me/bloughyt
My Tindie Store:
Ive started to sell stuff on Tindie, check it out!
https://www.tindie.com/stores/brianlough/
Other Platforms:
Twitter: @witnessmenow - https://twitter.com/witnessmenow
Instagram - https://www.instagram.com/brian_lough/
Instructables: witnessmenow - https://www.instructables.com/member/witnessmenow/
Support the Channel:
If you would like to support the channel, here are some ways you can.
1 - Like and comment on the videos - I really like hearing from you and apparently it helps!
2 - Share my videos - If you know somewhere that might be interested in my channel, please share it!
3 - Affiliate links - Where I get a small percentage of the purchase price but doesnt cost you anything extra.
Amazon.co.uk: http://amzn.to/2g7mJIO
Amazon.com : http://amzn.to/2ykDQ42
Amazon.de: https://amzn.to/2WMq7vF
Aliexpress: http://s.click.aliexpress.com/e/wdhojba
LCSC (where I buy most of my parts for Tindie stuff): https://lcsc.com/?promote=ODM5ZTB5NlkwNUN3bkNCZUtGb0ZlSWFKTGJIaTl3WThNWldwU0R6QndkZ3prNWdPemlTcG5XSi94U1Zpc0g5Y3MwYjFOUTNIZjJLbGhwWllsZ3N5QnR2WkZRdXMvdGVFcVhJWlJOdnNBV1U0RDZuY1M4TFlqZWNQYUo4RkN6VVJ2c0J4MTcvU3NEdw
4 - My Amazon store - Really its just a collection of stuff I use that I think are useful
https://www.amazon.co.uk/shop/brianlough
5 - Check out my Tindie Store
https://www.tindie.com/stores/brianlough/
My Tools & Kit*:
Ender 3 Pro 3d Printer - I got one of these recently and I love it!
- UK : https://amzn.to/31bebXT
- US : https://amzn.to/2IlSLid
Ts-100 Soldering Iron - I have this a couple of years and I love it, I thinking about getting a second!
- UK : http://amzn.to/2nlOF23
- US : http://amzn.to/2jDcyxl
Logitech C920 HD 1080p Pro webcam - I use one of these for my videos
- UK: http://amzn.to/2pnxk9Z
- US: http://amzn.to/2BsLMiF
Blue Snowball USB Microphone
- UK: http://amzn.to/2C0x4TH
- US: http://amzn.to/2DEA85J
Self Adjusting Wire Strippers
- UK: http://amzn.to/2C0urkR
- US: http://amzn.to/2BXQAh1
*= Affiliate Links")
