Big data is a term applied to data sets whose size or type is beyond the ability of traditional relational databases to capture, manage and process the data with low latency. Big data has one or more of the following characteristics: high volume, high velocity or high variety.
Big data analytics involves applying an algorithmic or mechanical process to derive insights and, for example, running through several data sets to look for meaningful correlations between each other. This allows analysts, researchers and business users to make better and faster decisions using data that was previously inaccessible or unusable.
Modin – multiprocess Dataframe tool with an identical API to pandas that allows users to speed up their Pandas workflows. Modin uses Ray or Dask to provide an effortless way to speed up pandas notebooks, scripts, and libraries.
Dask – open source library for parallel computing, which includes a task scheduling component for building dependency graphs and scheduling tasks as well as distributed data structures with APIs similar to Pandas Dataframes or NumPy arrays.
Koalas – tool, which has implemented pandas DataFrame API on top of Apache Spark and provides additional Koalas-specific functions to make it easy when working in a distributed setting.
Hadoop – software, that uses cluster computing to allow for faster data processing of large datasets using the MapReduce programming model.
Spark – data processing framework that can quickly perform processing tasks on very large data sets, and can also distribute data processing tasks across multiple computers, either on its own or in tandem with other distributed computing tools. This tool suits perfectly for working with big data in streaming form (real-time data, log data, API data), then Apache’s Spark is a great tool.
Pandas – tool , that allows to read data from many sources, create large dataframes (or matrixes / tables) from these sources and compute aggregate analytics based on what questions you’d like to answer. It has some built-in visualizations which can be used to chart and graph your results as well as several export functions to turn your completed analysis into an Excel Spreadsheet.
Numpy –instrument, that works perfectly large, multi-dimensional arrays and matrices with application of a large collection of high-level mathematical functions to operate on these arrays.