Coiled experienced two outages in the last few weeks; several production resources began operating too close to their limits, and our monitoring did not surface the risk early enough. The result was multiple production outages. This level o…
Nat Tabris and Alexandre Chabot-Leclerc Coiled makes it easy to take anything you can run locally and run it in the cloud. That way, you have access to more machines, bigger machines, or machines that are close to your cloud data. Many appl…
There are a lot of ways to run Python on AWS. You might use Lambda for event-driven tasks, Glue for managing data pipelines, or SageMaker for machine learning workloads. Each of these tools offers helpful abstractions, but also comes with c…
There have been a number of engineering improvements to Dask Array like consistent chunksizes in Xarray rolling-constructs and improved efficiency in map_overlap. Notably, as of Dask version 2024.11.2, calculating quantiles is much faster a…
Update Jan. 9th, 2025: We’ve added documentation for Coiled Batch. SLURM and other job schedulers have made it easy to run scripts on HPC systems for decades. Job scripts are straightforward and solve a common problem in a way that is famil…
Stephen Schneider, Franco Bosetti In this article, we showcase how Coiled technology enabled our data engineering and analytics team at Siemens to tackle complex data processing challenges. For an internal customer, we needed to analyze a l…
Sarah Johnson, James Bourbeau, Quentin Lhoest, Daniel van Strien This post was originally published on the Hugging Face blog. The Hugging Face platform has many datasets and pre-trained models that make using and training state-of-the-art m…
Hendrik Makait, Sarah Johnson, Matthew Rocklin We run benchmarks derived from the TPC-H benchmark suite on a variety of scales, hardware architectures, and dataframe projects, notably Apache Spark, Dask, DuckDB, and Polars. No project wins.…
People often ask “How does Dask compare to Spark?” Apache Spark has been the industry standard for big data, although it’s not clear that anyone especially loves it, especially in the Python space. The promise of Spark is that it won’t topp…
Running jobs on the cloud is common when processing cloud-hosted data, needing hardware that’s not available locally, or wanting to run in parallel across many machines. However, it’s surprisingly difficult to run basic jobs using APIs like…
Every day, grocers are tasked with predicting the future: they have to predict how many items will sell in the future to order the correct amount of inventory today. Since grocery retailers sell fresh food, which expires within a few days o…
GPUs are powerful but only have an impact when they’re easy to use 400 mins on CPU 85 mins on Apple M1 GPU GPUs can provide impressive performance boosts for certain workflows like training ML models, computer vision, analytics, and more. H…