Dask for huge datasets

Data Science, AI and ML
#1

Dask exposes low-level apis to its internal task scheduler to execute advanced computations. this enables the building of personalised parallel computing system which uses the same engine that powers dask’s arrays, dataframes, and machine learning algorithms.

what made dask tick
dask emphasizes the following virtues:

the ability to work in parallel with numpy array and pandas dataframe objects
integration with other projects.
distributed computing
faster operation because of its low overhead and minimum serialisation
runs resiliently on clusters with thousands of cores
real-time feedback and diagnostics