{"id":25545,"date":"2020-02-03T09:56:14","date_gmt":"2020-02-03T14:56:14","guid":{"rendered":"https:\/\/intelligencecommunitynews.com\/?p=25545"},"modified":"2020-02-03T10:00:21","modified_gmt":"2020-02-03T15:00:21","slug":"dataops-the-pursuit-of-simplicity","status":"publish","type":"post","link":"https:\/\/intelligencecommunitynews.com\/dataops-the-pursuit-of-simplicity\/","title":{"rendered":"DataOps, The Pursuit of Simplicity"},"content":{"rendered":"\n

Enterprise-scale data management and engineering best\npractices in support of advanced analytics<\/em><\/p>\n\n\n\n

By Mel Richey, Advanced Technology Manager, NT Concepts<\/p>\n\n\n\n

The Intelligence Community (IC) has placed highest\npriority on the continuous delivery of valuable analytic insights, whether\nderived from traditional analytic methodologies, AI and machine learning (ML),\nor human analysis. To support that mission, IC agencies need to operationalize\nthe data stream, make data discoverable with trusted lineage, and treat data\nitself as an asset much like analysis. This is the process called \u201cDataOps.\u201d <\/p>\n\n\n\n

Currently, data is trapped in silos across the\nenterprise. It is also unlabeled, stored in disparate formats, and not\ntransformed to be conducive for ingest into advanced analytic pipelines. DataOps\nis a process which removes the barrier to entry for analysts and technologists for\ndata to building advanced analytic solutions. \u201cOur customers have told us they\nwant to \u2018reduce the cost of curiosity,\u2019\u201d said Justin Nall, vice president of\nIntelligence at NT Concepts. \u201cOperationalizing data as an asset enables this\nfor analysts and technologists; unlocking data that already exists in their\norganization to solve mission problems.\u201d<\/p>\n\n\n\n

Reduce complexity <\/h3>\n\n\n\n

\u201cSimple can be harder than complex. You have to work hard\nto get your thinking clean to make it simple. But it\u2019s worth it in the end\nbecause once you get there, you can move mountains.\u201d \u2013 Steve Jobs<\/p>\n\n\n\n

For advanced analytics, software development, or AI\/ML\nsolutions, the most foundational elements are data. Many analytical missions\nare slowed by data discovery, that is, just figuring out what you have.<\/p>\n\n\n\n

To be useful, data need to be three things: <\/p>\n\n\n\n

1. Discoverable<\/strong>. Data needs to be tagged (e.g.,\ncommon or similar metadata tags) and stored in common formats (e.g., Excel\nfiles, json, delimited file formats, etc.). <\/p>\n\n\n\n

2. Accessible<\/strong>. Data needs to be easily callable\nand pullable from their respective storage locations without heavy access\ncontrols (to the extent possible), egress charges, or bandwidth limitations. <\/p>\n\n\n\n

3. Intelligible<\/strong>. Data needs to be internally\nformatted in accordance with an intelligible schema such that when a data file\nis opened, its contents are comprehensible.<\/p>\n\n\n\n

Despite these simple data governance standards, there are\nseveral ways in which the data themselves can be incorrect. Custodians of enterprise-scale\ndatasets or newly generated datasets should ensure that their data are:<\/p>\n\n\n\n