You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Creating a subdirectory for each neuron becomes an issue with larger models #8
At least on linux, the filesystem isn't really built for 10000s of subdirectories.
You run into issues like the ls command taking 10 seconds to run or taking minutes to get the size of the directory.
This is a pretty big problem when running this on large models is already a large undertaking.
I would suggest one of two solutions
A flat file structure where all graph files are in the same directory (maybe one subdirectory for each type of graph file) in order to minimize the number of subdirectories. I'm not actually sure this would fix the problems I mentioned. This should be tested.
Allow grouping the graphs into single files, e.g. tar balls. This could be configured by the user. Graphs could be grouped by model, layer og just by 512 neurons or something.
changed the title [-]Creating a subdirectory for each neuron becomes and issue with larger models[/-][+]Creating a subdirectory for each neuron becomes an issue with larger models[/+]on Nov 1, 2023
Activity
[-]Creating a subdirectory for each neuron becomes and issue with larger models[/-][+]Creating a subdirectory for each neuron becomes an issue with larger models[/+]