Understanding Data

Understanding Data 

Data is a distinct piece of information that is gathered and translated for some purpose. Data can come in the form of text, observations, figures, images, numbers, graphs, or symbols. For example, data might include individual prices, weights, addresses, ages, names, temperatures, dates, or distances.


Types of Data

  • Structured 

  • Unstructured

  • Semi-Structured

Structured Data -  Data that is organized in a defined manner or schema, typically found in relational databases. 

  • Characteristics: • Easily queryable • Organized in rows and columns • Has a consistent structure 

  • Examples: • Database tables • CSV files with consistent columns • Excel spreadsheets


Unstructured Data - Definition: Data that doesn't have a predefined structure or schema. 

  • Characteristics: • Not easily queryable without preprocessing • May come in various formats 

  • Examples: • Text files without a fixed format • Videos and audio files • Images • Emails and word processing documents


Semi -Structured Data - Data that is not as organized as structured data but has some level of structure in the form of tags, hierarchies, or other patterns. 

  • Characteristics: • Elements might be tagged or categorized in some way • More flexible than structured data but not as chaotic as unstructured data 

  • Examples: • XML and JSON files • Email headers (which have a mix of structured fields like date, subject, etc., and unstructured data in the body) • Log files with varied formats


Properties of Data 

  • Volume 

  • Velocity 

  • Variety


Volume - Refers to the amount or size of data that organizations are dealing with at any given time. 

  • Characteristics: • May range from gigabytes to petabytes or even more • Challenges in storing, processing, and analyzing high volumes of data 

  • Examples: • A popular social media platform processing terabytes of data daily from user posts, images, and videos. • Retailers collecting years' worth of transaction data, amounting to several petabytes.


Velocity - Refers to the speed at which new data is generated, collected, and processed. 

  • Characteristics: • High velocity requires real-time or near-real-time processing capabilities • Rapid ingestion and processing can be critical for certain applications 

  • Examples: • Sensor data from IoT devices streaming readings every millisecond. • High-frequency trading systems where milliseconds can make a difference in decision-making.


Variety - Refers to the different types, structures, and sources of data. 

  • Characteristics: • Data can be structured, semi-structured, or unstructured • Data can come from multiple sources and in various formats 

  • Examples: • A business analyzing data from relational databases (structured), emails (unstructured), and JSON logs (semi-structured). • Healthcare systems collecting data from electronic medical records, wearable health devices, and patient feedback forms.


Managing Data - Data Warehouse, Data Lake & Data Lakehouse


Data Warehouse : A centralized repository optimized for analysis where data from different sources is stored in a structured format. 

  • Characteristics:  Designed for complex queries and analysis • Data is cleaned, transformed, and loaded (ETL process) • Typically uses a star or snowflake schema • Optimized for read-heavy operations 

  • Examples: • Amazon Redshift • Google BigQuery • Microsoft Azure SQL Data Warehouse


Data Lake : A storage repository that holds vast amounts of raw data in its native format, including structured, semi-structured, and unstructured data. 

  • Characteristics: Can store large volumes of raw data without predefined schema • Data is loaded as-is, no need for preprocessing • Supports batch, real-time, and stream processing • Can be queried for data transformation or exploration purposes 

  • Examples: Amazon Simple Storage Service (S3) when used as a data lake • Azure Data Lake Storage • Hadoop Distributed File System (HDFS)


Data Lakehouse : A hybrid data architecture that combines the best features of data lakes and data warehouses, aiming to provide the performance, reliability, and capabilities of a data warehouse while maintaining the flexibility, scale, and low -cost storage of data lakes. 

  • Characteristics: • Supports both structured and unstructured data. • Allows for schema-on-write and schema-on-read. • Provides capabilities for both detailed analytics and machine learning tasks. • Typically built on top of cloud or distributed architectures. • Benefits from technologies like Delta Lake, which bring ACID transactions to big data. 

  • Examples: • AWS Lake Formation (with S3 and Redshift Spectrum) • Delta Lake: An open-source storage layer that brings ACID transactions to Apache Spark and big data workloads. • Databricks Lakehouse Platform: A unified platform that combines the capabilities of data lakes and data warehouses. • Azure Synapse Analytics: Microsoft's analytics service that brings together big data and data warehousing

Comments

Post a Comment

Popular posts from this blog

How do I Use the Linux Top Command?

The Unix top command is designed to help users determine which processes are running and which applications are using more memory or processing power than they should be. The top command is very easy to use but you should know the things in details. The output of to is : top output: top - 22:09:08 up 14 min, 1 user, load average: 0.21, 0.23, 0.30 Tasks: 81 total, 1 running, 80 sleeping, 0 stopped, 0 zombie Cpu(s): 9.5%us, 31.2%sy, 0.0%ni, 27.0%id, 7.6%wa, 1.0%hi, 23.7%si, 0.0%st Mem: 255592k total, 167568k used, 88024k free, 25068k buffers Swap: 524280k total, 0k used, 524280k free, 85724k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 3166 apache 15 0 29444 6112 1524 S 6.6 2.4 0:00.79 httpd 3161 apache 15 0 29444 6112 1524 S 5.9 2.4 0:00.79 httpd 3164 apache 15 0 29444 6112 1524 S 5.9 2.4 0:00.75 httpd 3169 apache 15 0 29444 6112 1524 S 5.9 2.4 0:00.74 httpd 3163 apache 15
Read more

IOPS measurement

Image
IOPS (input/output operations per second) is the standard unit of measurement for the maximum number of reads and writes to non-contiguous storage locations of devices like hard disk drives (HDD), solid state drives (SSD), and storage area networks (SAN). IOPs is often measured with an open source network testing tool called an Iometer. Some commonly accepted averages for random IO operations, calculated as 1/(seek + latency) = IOPS:  IOPS calculations Every disk in your storage system has a maximum theoretical IOPS value that is based on a formula. Disk performance — and IOPS — is based on three key factors: Rotational speed (aka spindle speed). Measured in revolutions per minute (RPM), most disks you'll consider for enterprise storage rotate at speeds of 7,200, 10,000 or 15,000 RPM with the latter two being the most common. A higher rotational speed is associated with a higher performing disk. This value is not used directly in calculations, but it is highly i
Read more