Convert & Import Data into InfluxDB
When using Telegraf it's easy enough to insert data into Influx and visualize it with Grafana - but what if you have metrics that Telegraf doesn't have the capacity to pull and report to Influx?
In this example I'm using uBot to scrape several metrics into a .txt file that I want to insert into Influx. It doesn't matter how you scrape the, whether you use VBScript, Python or uBot, as long as that data is written to a specially formatted .txt file.
In my particular case, I'm using uBot to scrape metrics from my power company and from my ISP to give me the following metrics:
- Power company pay period
- Kilowatt usage to date
- Amount due (if any)
- ISP amount due (if any)
- Data usage to date
Once this data has been scraped, it's written to a text file in a specific format that I will cover below.
Click here for a tutorial on how to use uBot to scrape metrics into this specifically formatted text file.
This is a Windows specific tutorial.
This tutorial might be compatible with Linux/other platforms when you remove the Unix/Windows (ANSI/UTF-8) file type conversion process.
- Notepad (or other text editor)
1. The text file's format
This tutorial assumes that you have already scraped metrics from some source (Click here for a tutorial on how to scrape metrics with uBot). The format for the .txt file is as follows:
# DML\n # CONTEXT-DATABASE: telegraf dataUsage,host=att value=29 ATT_Due,host=att value=0 gulfPowerDue,host=gulfpowerDue value=469.54 gulfPowerKWh,host=gulfpowerKWh value="3,540" gulfPowerPeriod,host=gulfpowerPeriod value="6"
I outputted the above .txt file from uBot and saved it on my hard drive as "data.txt", and this is what it looks like on my Windows system:
2. Using Powershell to convert data.txt's file format from Windows to Unix
Influx will not accept a Windows formatted text file. It must be converted from ANSI (Windows) to UNIX (UTF-8). You can use a Powershell script to do that, so copy and paste the commands below and save them into a Powersshell executable file, which is a text file with the extension ".ps1". In my case, I've saved the file as "convert_to_unix.ps1".
Be sure to change the "data.txt" to reflect your data file's name and location:
$original_file ='C:\Users\USERNAME\Documents\influxdb-1.7.3-1\data.txt' $text = [IO.File]::ReadAllText($original_file) -replace "`r`n", "`n" [IO.File]::WriteAllText($original_file, $text)
This is what my saved Powershell script looks like on my Windows system:
3. Create the Batch file that fires the Powershell conversion script and the InfluxDB insert commands
Now we will combine the Powershell UNIX conversion script into a BATCH file that, in sequence, converts the "data.txt" file into UNIX format and then immediately calls Influx to import data.txt:
powershell.exe -executionpolicy bypass -file C:\Users\USERNAME\Documents\influxdb-1.7.3-1\convert_to_unix.ps1 influx -import -path=data.txt
And this is what the command looks like after it has successfully been processed:
At this point these metrics will become available from within Grafana: