The Lost Art of Data Profiling
Bob Conway, Information Engineering Associates
Data profiling is the technique of inferring the true semantic meaning of data by carefully scrutinizing the structure and content of existing databases. The title refers to data profiling as an art because it is much more than mechanically capturing distinct values and min/max values of data elements. Data profiling is the heuristic interpretation of those statistics to discover the underlying business rules. Data profiling is true reverse engineering.
This bottom- up method of data modeling is a valuable complement and validation of top down modeling methods such as user interviews and group consensus (JAD) sessions. Data profiling is also the most reliable way to identify data quality issues and design ETL procedures to mitigate them.
Bob Conway is the founder of and Principal Consultant with Information Engineering Associates, a Colorado-based consulting firm focused on training and development of data management best practices. His twenty-five year consulting experience spans diverse industries including, Oil & Gas, Financial Services, Manufacturing, Retail, Automotive, Telecommunications, Health Care, Federal and Municipal Government. Bob has implemented successful data warehouses for numerous clients from small start-ups to Fortune 500 firms with demonstrable return-on-investment. As an adjunct faculty member Bob has taught graduate level Information Management classes at two major universities. Bob is a frequent speaker at local and national conferences. He brings his rich business and technical experience into presentations with real world examples. His delivery style is informative and entertaining.