Overview: Create a dependency diagram using pipforester and pipdeptree

[stevepiercy]

Description

Newbies to Plone have a tough time navigating between all the various packages that make up Plone.

A Python dependency diagram would be helpful for the Python parts. We'll use https://pypi.org/project/pipforester/ which depends on https://pypi.org/project/pipdeptree/.

Humans can fill in any important bits, such as "To edit breadcrumbs or other bits and pieces of a layout in Plone core, then visit plone.app.layout."

For Volto, do we already have a tool that can do this? A quick search turned up https://github.com/sverweij/dependency-cruiser

The appropriate place for this graph and any associated narrative would be in a new section in Overview.

[ksuess]

pydeps looks like a pretty cool enhancement. I remember that @jensens was or is still working on disentangling dependencies. Which are your tools, @jensens?

[stevepiercy]

This might be closed by #1715 when it is merged.

[stevepiercy]

@ksuess @jensens @mauritsvanrees OMG, check this out! Zero configuration.

venv/bin/pip install pydeps
venv/bin/pydeps submodules/plone.api/src/plone

Would this be helpful at all? I have no idea how difficult it is to update the Mermaid dependency graph.

[jensens]

Looks nice. But, I have no idea. Lines without meaning? What does it tell us?

If you want a dependency tree of Plone as defined in pyproject.toml (or legacy in setup.cfg/-.py), I suggest to use pipdeptree/pipforester. In a Virtual Environment with Plone installed do:

# debian/ubuntu: 
sudo apt install libgraphviz-dev xdot
# install tools
pip install pipdeptree pipforester
# create tree 
pipdeptree -j > plone.json
# prune tree
pipforester -i plone.json -o plone.dot 

This creates a huge graph. To focus on Plone several excludes could be added to pipdeptree like so:

pipdeptree -j -e 'zope.*,z3c.*,zc.*,setuptools,uv,six' > plone.json 
pipforester -i plone.json -o plone.dot 

This shows you (dot -Tpng -o plone.png plone.dot) something like this (from a customer project I am currently working on, so better do this on a vanilla Plone - and it needs more excludes and less test extras):

See also https://pypi.org/project/pipforester/

[jensens]

Would this be helpful at all? I have no idea how difficult it is to update the Mermaid dependency graph.

The Mermaid graph has different focus: Overview. Both, your initial graph and the pipdeptree/pipforester are going too much into detail.

[stevepiercy]

Thanks for the explanation. I tried to view your diagram on a 39-inch display, and it's still a rat's nest, especially for the Plone deps. That is way too much visual information to be useful.

After reading about how you generate diagrams with pipforester, I did a little more digging.

From pydeps notes:

pydeps finds imports by looking for import-opcodes in python bytecodes (think .pyc files). Therefore, only imported files will be found (ie. pydeps will not look at files in your directory that are not imported). Additionally, only files that can be found using the Python import machinery will be considered (ie. if a module is missing or not installed, it will not be included regardless if it is being imported). This can be modified by using the --include-missing flag.

Whereas pipforester analyzes the results from pipdeptree, which in its readme says:

pipdeptree is a command line utility for displaying the installed python packages in form of a dependency tree.

So I think that this issue should be renamed to using pipforester and pipdeptree, because it seems that pydeps does not do what we want whereas pipforester does. Would you agree?

If so, then we can use your notes as a starting point, and try to find a reasonable dependencies diagram.