Scraping ISP Data Usage With uBot Studio

bot_01.jpg

This is a pretty easy way of scraping data from a website, but it does cost about $300. You can accomplish the same thing in Python, the catch being is that you have to know Python... so for those of you who don't, uBot is great.

In this tutorial I will be using AT&T as an example, but with some adjustment it should be easy enough to adapt it for any ISP that gives you a running total of your monthly data usage.

By the end of this tutorial you will have a bot that can be fire automatically to log into your AT&T control panel, scrape the current data usage, and write that data into a file called "data.txt" on your computer's hard drive.

You MUST convert the data.txt file to UNIX before InfluxDB will accept it.

This is the first of four sub-tutorials that will enable you to pull data from a web page and post it into InfluxDB.

This is a Windows specific tutorial.

Once you have completed this tutorial, please see the following tutorial:

Dependencies:

  • None

1. Download uBot Studio

2. Create the uBot script

Since I'm using the Community edition of uBot studio, I can't share my projects or export the script to an executable file, but I can give you the steps that I used to pull my data usage.

In this example, when I began creating the bot, I had already opened uBot, navigated to AT&T and logged in (https://www.att.com/my/#/login). This reason for this is that uBot's built in Chromium browser will remember my e-mail address, a feature that we can use to make the script shorter. Once you've manually visited and logged into AT&T, log out again.

  • NAVIGATE: https://www.att.com/my/#/login
  • bot_02.JPG
  • WAIT: 5 seconds
  • bot_03.JPG
  • CLICK: on the saved e-mail address
  • bot_04.JPG
  • WAIT: 5 seconds
  • bot_03.JPG
  • TYPE TEXT: In the password field
  • bot_05.JPG
  • WAIT: 5 seconds
  • bot_03.JPG
  • CLICK: the "sign in" button
  • bot_06.JPG
  • WAIT: 30 seconds (this page loads notoriously slowly)
  • bot_07.JPG
  • SET: variable - this is where we will be scraping the text between HTML elements "usageText"><strong>" and "</strong>" and saving that data as a variable
  • bot_08.JPG
  • WAIT: 5 seconds
  • bot_03.JPG
  • SAVE TO FILE: This is where we are going to write the variable to a text file on your hard drive, and this text file will be formatted as data that is friendly to be imported into InfluxDB.
  • bot_09.JPG
  • WAIT: 1 second
  • bot_10.JPG
  • CLICK: the profile button - this is the drop down menu that probably has your AT&T username on it.
  • bot_11.JPG
  • WAIT: 5 seconds
  • bot_03.JPG
  • CLICK: the sign-out button
  • bot_11.JPG

At this point you should now have a text file on your hard drive titled "data.txt". This is the file that you can import into InfluxDB via the command line, however you CANNOT import this file in its current state on a Windows computer - you MUST convert it to UNIX format before InfluxDB will accept it.

You MUST convert the data.txt file to UNIX before InfluxDB will accept it.

Once you have completed this tutorial, please see the following tutorial:

 

 

 


Revision #7
Created Thu, Sep 19, 2019 8:21 PM by Cameron
Updated Tue, Oct 8, 2019 5:03 PM by Cameron