Monday, March 9, 2020

Code: Read an adl:// stream in Python

Edit: I'm now putting code snippets on my GitHub Pages site here.

I recently had the need to read in an adl:// file in some Python code I was working on. Dask has this capability, and most search results for reading ADL streams in Python point you to Dask itself. But I wanted to avoid using Dask for this.

I couldn't find anything that said how you can do this directly in Python, but it turns out Dask just wraps fsspec, and the code to do it using fsspec directly is pretty easy: