'Judge Dee's Mystery' on Netlfix is a treat for fans of van Gulik's Judge Dee novels, and it's also a stunningly beautiful production.

I now have the Semantic Scholar dataset importer running with PostgreSQL on a Raspberry Pi model 5 and pimoroni nvme base using Crucial Memory. Goodness, it's fast!

I will be documenting my work on a Python tool for populating and using a local copy of the Semantic Scholar Academic Graph corpus here:
romillycocking.substack.com/subscribe?utm_…

Copying 0.5 TB of data from pi5 to pi5. The target machine has a 2TB Crucial Memory nvme p3+ board with a pimoroni nvme hat.
In the 80s a system that held 2 TB was a major purchase. Now that power is available for < £200.

Inserting a large dataset into Postgres with indices is much slower than loading w/o indices and adding them later.
The plot below shows the time to load batches of 100K records into a table with/without indices.
Without, the time per batch hardly changes.

More work on importing Semantic Scholar datasets into Postgres. I just inserted 13,611,218 citation rows in 6 min, so it looks as if everything will be possible!

I'm still tuning my Semantic Scholar dataset loader with some remarkable results. More news tomorrow.

I'm gathering all the #Python on hardware news for the week - if you have any please send to cpnews (at) adafruit (period) com. And I hope you're subscribed to get this free newsletter every Monday. No spam, your email isn't used for anything else, leave easily anytime. Thanks!

I've had to delay my write-up about loading 1 TB of Semantic Scholar data into PostgreSQL on a Raspberry Pi. I have resolved more issues and I'm now trying a final speedup to the import process.
I'll write up my experiments this weekend.