Query planning for Dask Dataframe released

github.com/dask/dask/issu…

Dask dataframe is pretty fast now, even when folks write less-than-perfect code.

$ pip/conda/uv/mamba install dask

Join us next week to see how to schedule Python jobs with Prefect and Coiled.

Prefect makes it easy to write production workflows in Python. Coiled makes it easy to deploy Prefect in the cloud.

Register for the webinar: content.coiled.io/prefect-webinar

Join us to see how to churn through many cloud files in parallel. We’ll cover:
- Adapting existing code with minimal changes
- Cost optimization
- Auto-scaling to process TBs of data in parallel

🗓 Wednesday, January 31st
⏰ 11 am Eastern
Register: content.coiled.io/parallel-cloud…

New blog post on how to schedule Python jobs with Prefect + Coiled

📢Really happy to announce the 'Scaling with Dask guide' for Panel.

🤝In many ways Panel and Dask is a perfect couple as they share both philosophies and key technologies.

👇Link to guide below

#python #dataviz pandas Coiled #datascience #Analytics

New blog post from carbonplan on how they used Kerchunk + Coiled to speed up their climate data workflow by 10x.

Blog post: carbonplan.org/blog/kerchunk-…

Code: github.com/carbonplan/ker…

Coiled will be at  #AGU23 next week! We'll be sharing a space with Pangeo and Earthmover, next to Google.

We'll have sofas, snacks, and demos on processing terabyte-scale cloud datasets
youtube.com/watch?v=rDvTqo…

Join us tomorrow to see how to run your Python code in the cloud with Amazon Web Services. We'll have live demos showing how to process terabyte-scale cloud data.

🗓 Thursday, Nov. 30th
⏰ 1pm Eastern
Register here pages.awscloud.com/easily-run-you…

Join us for tomorrow to learn more about scaling NASA Earthdata workflows in the cloud with National Snow and Ice Data Center and Openscapes @[email protected].

🗓 Tuesday, Nov. 28th
⏰ 1pm ET
Register here: openscapes.org/events/2023-11…

Join us for live demos + conversation with National Snow and Ice Data Center and Openscapes @[email protected]. There will be:
- Cloud access and infrastructure discussions
- Live demo processing NASA Earthdata in a big parallel for-loop in the cloud

🗓 Tuesday, Nov. 28th
⏰ 1pm ET
Register here: openscapes.org/events/2023-11…

New blog post on how to process large datasets with polars data in the cloud.

Polars is a great tool for querying large datasets. When you need more memory, you can use Coiled serverless functions to run Polars on a big VM in the cloud. medium.com/coiled-hq/proc…

Join us for our webinar on how Dask DataFrame is fast now
- TPC-H benchmarks for Dask, Spark, DuckDB, Polars
- How Dask expressions optimizes your code (like Spark SQL would)
- Use GPUs with RAPIDS AI

🗓 Thursday, Nov. 16th
⏰ 11am ET
Register here: content.coiled.io/dask-expr-webi…

New blogpost: Processing Terabyte-scale NASA cloud datasets:
medium.com/coiled-hq/proc…

Alternate title: 'We built a fancy task scheduler and all NASA wanted was this parallel for loop'

It's been fun working with Openscapes @[email protected] to migrate NASA data workloads to the cloud. Recently, a group from National Snow and Ice Data Center demoed how they're using Coiled Functions to process a large altimetry dataset from NASA's ICESat-2 mission openscapes.org/blog/2023-10-1…