User Tools

Site Tools

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
mission:log:2014:11:24:using-python-lxml-request-as-simple-scrape-robot-for-metrics-from-webpages [2015-07-08 08:07] – [In the beginning there was the copy] chronomission:log:2014:11:24:using-python-lxml-request-as-simple-scrape-robot-for-metrics-from-webpages [2016-08-09 19:13] (current) – Updated VFCC links chrono
Line 30: Line 30:
 Unfortunately the data isn't accessible through an API or at least some JSON export of the raw data. Which meant I needed to devise a robot that would periodically scrape the data from that web page, extract all needed values and feed that data into the UCSSPM to calculate with real data for reference. Once it has done all that it has to push all usable raw data and the results of the UCSSPM prediction into an influxdb shard running on the stargazer so that the data can be stored, queried and (re)viewed live on the following VFCC dashboards: Unfortunately the data isn't accessible through an API or at least some JSON export of the raw data. Which meant I needed to devise a robot that would periodically scrape the data from that web page, extract all needed values and feed that data into the UCSSPM to calculate with real data for reference. Once it has done all that it has to push all usable raw data and the results of the UCSSPM prediction into an influxdb shard running on the stargazer so that the data can be stored, queried and (re)viewed live on the following VFCC dashboards:
  
-  * [[https://apollo.open-resource.org/flight-control/vfcc/#/dashboard/db/aquarius-external-environment|External Environment Data]] +  * [[https://apollo.open-resource.org/flight-control/vfcc/dashboard/db/aquarius-external-environment|External Environment Data]] 
-  * [[https://apollo.open-resource.org/flight-control/vfcc/#/dashboard/db/aquarius-solar-power|Aquarius Solar Power]] +  * [[https://apollo.open-resource.org/flight-control/vfcc/dashboard/db/aquarius-solar-power|Aquarius Solar Power]] 
-  * [[https://apollo.open-resource.org/flight-control/vfcc/#/dashboard/db/odyssey-solar-power|Odyssey Solar Power]]+  * [[https://apollo.open-resource.org/flight-control/vfcc/dashboard/db/odyssey-solar-power|Odyssey Solar Power]]
 ===== The bash solution ===== ===== The bash solution =====
  
Trace:

Page Tools