Conversation
awkward-pandas is deprecated. Using PyArrow instead as described in https://awkward-array.org/doc/main/user-guide/how-to-convert-pandas.html . This does change the output data type slightly, see tests affected by this commit.
|
Note: The predecessor of awkward-pandas is akimbo (https://github.com/intake/awkward-pandas even forwards to https://github.com/intake/akimbo). That is a library with the feature that as soon as you write |
Codecov Report❌ Patch coverage is
❌ Your patch check has failed because the patch coverage (78.57%) is below the target coverage (98.00%). You can increase the patch coverage or adjust the target coverage. Additional details and impacted files
|
|
Just to remark this: The codecov error stems from the fact that the tests do not test what happens when a module is not installed. However I believe this to be usual for these tests. |
|
Hi @2xB, thank you for this PR and sorry for the delay! I'm a bit concerned with some of the changes since dtype info is completely lost, and also since there are intermediate steps as Python lists, which could have really bad performance when working with large datasets. From what I understand, |
|
Hej @ariostas, oh, I used the wrong word. Anyways, intake/akimbo#52 removed We can leave out the I just added the explicit Also, I added a dependency on |
|
Note: The HDF5 export issue from #1706 is no longer fixed since using |
|
Ah, but before #734 there were no extension arrays and HDF5 export was not an issue. Back in the days, no |
There was a problem hiding this comment.
Thank you, @2xB! I like this a lot better than the previous version. Unfortunately, there would still be some behavior changes, but I think this is probably the best option that we have since awkward_pandas is deprecated. I think it should be fine since I don't think as many people use the pandas side, but I want to discuss this a bit with other people to see what they think.
I've attached some comments from Claude below.
🤖 AI text below 🤖
Thanks for the update! Moving to ArrowExtensionArray + akimbo is a big improvement over the earlier object-dtype version: columns stay columnar, and .ak works again. I ran the pandas-related tests locally on this head (108 passed, 1 skipped for needing ROOT; network/xrootd/distributed not run) and left inline comments. The points that don't belong on a single line:
This is a breaking change. The title is fix: and the akimbo commit is feat: without !, so release tooling would treat it as a patch/minor bump. For users:
- the column dtype changes from
awkwardto an Arrow dtype; - cells (
s[i]) are now Python lists instead ofak.Array(the updatedtest_0912asserts this); - arithmetic directly on the Series (
s * 2) no longer works (s.ak * 2does); uproot.extras.awkward_pandas()is removed.
Could you retitle to feat!: (or add a BREAKING CHANGE: footer) and include a short migration note?
PR description is out of date. It still mentions "Python arrays of NumPy arrays" and awkward.array_equal, which no longer match the code. It also says this no longer fixes #1706, so that issue should stay open.
|
|
||
| def finalize(self, array, branch, interpretation, entry_start, entry_stop, options): | ||
| pandas = self.imported | ||
| uproot.extras.akimbo_pandas() # Automatically adds .ak accessors to Pandas DataFrames |
There was a problem hiding this comment.
🤖 AI text below 🤖
This makes akimbo a hard requirement for every library="pd" call, including flat numeric data. With akimbo not installed:
events.arrays(["NMuon", "MET_px"], library="pd")
# ModuleNotFoundError: install the 'akimbo' package with: ...On main, flat branches need only pandas. Could this import be optional (try to import akimbo.pandas, and continue without it if it's missing), or moved into the jagged branch of _process_array_for_pandas? It also runs once for every branch that gets finalized.
Making it optional also helps with two other points:
- akimbo's last PyPI release is 2025.3.0 (conda-forge only has 2024.12.0), so it's better not to make it a hard dependency.
- awkward-pandas and akimbo both register
.ak, and whichever is imported last wins. If a user still has awkward-pandas installed and imports it after uproot,s.akfails with "ak accessor called on incompatible data". It's worth telling users in the docs to uninstall awkward-pandas.
There was a problem hiding this comment.
As said in the comment above, I don't really know if I want akimbo a dependency at all. Yes, it brings back the .ak accessor, but it does so by altering every Pandas dataframe that ever exists in that session. I would definitely not make it optional and automatically import if it is available since that means code using the .ak accessor would fail with confusing error messages that point to everything but not that the users should just install this random Python module.
| pandas = uproot.extras.pandas() | ||
| uproot.extras.pyarrow() # The next line requires PyArrow | ||
| array = pandas.arrays.ArrowExtensionArray( | ||
| awkward.to_arrow(array, extensionarray=False) |
There was a problem hiding this comment.
🤖 AI text below 🤖
With extensionarray=False, Awkward-specific types are lost when converting back with s.ak.array:
uproot-HZZ-objects.root:events/muonp4:var * TLorentzVector[fP: TVector3[...], fE: float64]becomesvar * {fP: {fX, fY, fZ}, fE}. The record names, and the behaviors attached to them, are gone.uproot-stl_containers.root:tree/map_int32_int16:var * tuple[[int32, int16], parameters={"__array__": "sorted_map"}]becomesvar * {"0": int32, "1": int16}, and cells print as[{'0': 1, '1': 1}, ...].
With extensionarray=True, both convert back to exactly the original type, and so do float32 and nullable ints. The trade-off is that pandas' own .list accessor doesn't recognize the extension type, but akimbo's .ak covers that. I'd suggest extensionarray=True, since ROOT data is full of named records and maps.
There was a problem hiding this comment.
Quoting from https://awkward-array.org/doc/main/user-guide/how-to-convert-pandas.html#conversion-through-apache-arrow :
The extensionarray flag is normally True, and enables Awkward to preserve metadata through Arrow transformations. However, tools like Arrow’s Pandas conversion do not recognise Awkward’s special extension type, so we must take care to provide Arrow with native types: [...]
So either the documentation is wrong or outdated or this is not supported to my understanding.
There was a problem hiding this comment.
The documentation being outdated can btw. very well be, it does not use ArrowExtensionArrays at all. Or one is not supposed to use them in this conversion...
| else: | ||
| array = uproot.extras.awkward_pandas().AwkwardExtensionArray(array) | ||
| pandas = uproot.extras.pandas() | ||
| uproot.extras.pyarrow() # The next line requires PyArrow |
There was a problem hiding this comment.
🤖 AI text below 🤖
Minor: awkward.to_arrow already raises an ImportError with install instructions when pyarrow is missing, so this bare call (whose result is thrown away) can probably be removed.
|
|
||
|
|
||
| def awkward_pandas(): | ||
| def pyarrow(): |
There was a problem hiding this comment.
🤖 AI text below 🤖
uproot.extras.awkward_pandas() is public and is removed here. Downstream code that calls it would get AttributeError. Could it stay for a release, emitting a DeprecationWarning that points to pyarrow/akimbo?
There was a problem hiding this comment.
If downstream code depended on the uproot importer, that would be a very, very confusing design decision from my point of view. But even if it did because of Hyrum's law ( https://xkcd.com/1172/ ), I believe it is trivial to find this PR. I would prefer to not introduce stale code covering such a weird use case.
| # IndexError: too many indices for array: array is 1-dimensional, but 2 were indexed | ||
|
|
||
| The Pandas form for this type of data is a DataFrame with Awkward Dtype, provided by the `awkward-pandas <https://github.com/intake/awkward-pandas>`__ package. | ||
| The Pandas form for this type of data is a DataFrame with PyArrow dtype, provided by the `Pandas ArrowExtensionArray <https://pandas.pydata.org/docs/reference/api/pandas.arrays.ArrowExtensionArray.html>`__ type. |
There was a problem hiding this comment.
🤖 AI text below 🤖
The example output right below is now stale. With this PR, cells print as NumPy-style float32 values, e.g. [-52.899456 37.73778 ], not full-precision Python lists like [-8.16079330444336, -11.307581901550293]. It would be good to regenerate it.
There was a problem hiding this comment.
Ah so there is a precision loss. That's quite bad, I believe.
| [2421 rows x 6 columns] | ||
|
|
||
| You can operate on Awkward Array data in Pandas using the ``.ak`` accessor; see the [awkward-pandas documentation](https://awkward-pandas.readthedocs.io/en/latest/quickstart.html). | ||
| You can operate on Awkward Array data in Pandas using the ``.ak`` accessor; see the [akimbo documentation](https://akimbo.readthedocs.io/en/latest/quickstart.html). |
There was a problem hiding this comment.
🤖 AI text below 🤖
It would help to add a short migration note here, because cells are now Python lists rather than ak.Array. Arithmetic on a raw cell now fails silently instead of raising:
s[0] * 2 # before: <Array [3, 5]> now: [1.5, 2.5, 1.5, 2.5]
s[0] + s[0] # before: <Array [3, 5]> now: [1.5, 2.5, 1.5, 2.5]Suggested wording: to get an Awkward array for a row, use s.ak.array[i]. In loops, convert the column once (arr = s.ak.array, then arr[i]), because each s.ak.array call converts the whole column (about 0.3 ms on 1M rows).
There was a problem hiding this comment.
I think if behavior changed, this makes much sense. If you have a good wording, feel free to add this here, ideally mentioning Uproot version numbers - that is great. Thanks!
|
|
||
|
|
||
| def test_jagged_pandas(): | ||
| awkward = pytest.importorskip("awkward") |
There was a problem hiding this comment.
🤖 AI text below 🤖
Nit: awkward is imported here but never used in this test.
There was a problem hiding this comment.
Sure. The test only works if awkward is available due to the nature of this Pull Request, but I can skip assigning it to a value.
As remarked in the PR review.
As the PR review remarked, this is already tested for by the Awkward library.
|
Before we continue, I have to admit that the Claude comment on updating the quickstart guide is actually quite worrying to me. That looks like we have significant precision loss when using PyArrow. I think in a scientific environment, that is quite a bad regression. Intuitively I have no idea why this is, but if that is an issue occurring due to PyArrow, I would like to bring up again the option to revert #734 instead, I don't think that had such a precision loss. |
This branch has not been deployed
2 participants
awkward-pandas is deprecated. This PR solves that by using PyArrow instead as described in https://awkward-array.org/doc/main/user-guide/how-to-convert-pandas.html . This does change the output data type slightly, see tests affected by this commit.
Some arrays in the test are no longer plain nested Python arrays but Python arrays of NumPy arrays, using
awkward.array_equalto compare them across types.I tried to reflect the changed behavior in the documentation. This does change the dependency on awkward-pandas to a dependency on PyArrow, which I however believe to be justified since the above linked Pandas guide itself suggests doing so.
[edit] Since using
ArrowExtensionArray, no longer solving #1706[edit] Side note: I thought about whether to try and replicate the old behavior of awkward-pandas, but I believe it should be better to use the documented approaches. It is worth noting that having NumPy arrays instead of plain Python arrays or no longer having the
.akaccessor but instead a different code to get Awkward arrays does affect the usage of this library, so this commit could also be taggedBREAKING CHANGE:. Btw. Pandas nowadays also has a native ArrowDtype https://pandas.pydata.org/docs/reference/api/pandas.ArrowDtype.html, but that is still written as experimental. But going from the Awkward dtype of awkward-pandas to the ArrowDtype provided by Pandas would rely on experimental features and I do not know if one would gain much.