Data, undoubtedly, is the cornerstone in Data Science and AI. You need the right amount and quality of data to get your results right. And as the famous saying goes, “Garbage in = Garbage out”, we need to be certain of what the data is before working on it.
When it comes to collecting the data, there are mainly 3 broad steps:
-
Knowing the Data and the sources: When the business problem is identified, the relevant features in the data are gathered. The data could be either Structured (present in structured Databases) or Unstructured (text, image, etc.). Data could be internal or external to the company, with internal data being more reliable, but external data being equally important.
-
Data pipelines: Pipelines are built to query, filter and transform the data which could be in Batches, Streamed (real-time) or even Hybrid. Multiple tools are available for this and it is the job of Data Engineers to build the pipelines suiting to their data and business needs.
-
Storing the Data: With the pipelines ready, the data has to be ingested in the analytical environments to be processed, cleaned and explored to model. Storing can be On-Premise, on Cloud Databases or in Data Lakes, etc.
