Skip to main content

Managing a Million Data Points: A How-To Guide

Buy Article:

$17.50 plus tax (Refund Policy)

Or sign up for a free trial

Abstract:

The Hampton Roads Sanitation District (HRSD), serving the Tidewater Area of Virginia, has been tasked to maintain strict data reliability of its flow, pressure, and rain gauge monitoring sites. Per the Environmental Protection Agency (EPA) Administrative Order (AO), HRSD is required to maintain 75% data reliability for each individual meter during each monthly monitoring period and 90% data reliability for all data for each type of monitoring sensor or meter during qualifying wet weather events. HRSD has deployed 122 pressure monitoring sites, 175 flow monitoring sites and 66 rain gauge sites. Approximately 87,000 data points must go through a daily QA/QC process to ensure data validity and reliability for the flow meters and pressure sensors alone. The automation of data quality alerts through SQL statements has created a streamlined, efficient method to handle thousands of data points on a daily basis to identify system changes, system reaction, and instrument fouling.

Keywords: Data Analysis; SQL; Telog; wastewater

Document Type: Research Article

DOI: https://doi.org/10.2175/193864710798193446

Publication date: 2010-01-01

More about this publication?
  • Proceedings of the Water Environment Federation is an archive of papers published in the proceedings of the annual Water Environment Federation® Technical Exhibition and Conference (WEFTEC® ) and specialty conferences held since the year 2000. These proceedings are not peer reviewed.

    A subscription to the Proceedings of the Water Environment Federation includes access to most papers presented at the annual WEF Technical Exhibition and Conference (WEFTEC) and other conferences held since 2000. Subscription access begins 12 months after the event and is valid for 12 months from month of purchase. A subscription to the Proceedings of the Water Environment Federation is included in Water Environment Federation (WEF) membership.

    WEF Members: Sign in (right panel) with your IngentaConnect user name and password to receive complimentary access. Access begins 12 months after the conference or event
  • Subscribe to this Title
  • Membership Information
  • About WEF Proceedings
  • WEFTEC Conference Information
  • Learn about the many other WEF member benefits and join today
  • Ingenta Connect is not responsible for the content or availability of external websites
  • Access Key
  • Free content
  • Partial Free content
  • New content
  • Open access content
  • Partial Open access content
  • Subscribed content
  • Partial Subscribed content
  • Free trial content
Cookie Policy
X
Cookie Policy
Ingenta Connect website makes use of cookies so as to keep track of data that you have filled in. I am Happy with this Find out more