Error in calculating preservation statistics

[LiuCanidk]

Hi, thanks for developing this nice tool.

I encountered an error when running function calculatePresStats following the tutorial script in the CroCoNet-analysis repository (part 3. brain dataset).

pres_stats_jk <- calculatePresStats(pruned_modules, 
                                    network_list, c("cor_adj", "cor_kIM"), 
                                    replicate2species, n_cores = 5)

Error:
task 1 failed - "could not find function "vectorize"

Could you please help figure this out?

[LiuCanidk]

I fixed this bug by changing the source code of calculatePresStats:

trace(CroCoNet::calculatePresStats, edit=T)

In line 90, change vectorize to as.vector

However, I encountered another error about calculating cor_kIM (the bug of cor_adj mentioned above has been fixed):

pres_stats_jk <- calculatePresStats(pruned_modules, 
                                    network_list, c("cor_adj", "cor_kIM"), 
                                    replicate2species, n_cores = 16)

Output:
Calculating cor_adj using weight*direction
Error in { :
task 10 failed - "ℹ In argument: cor_kIM = unname(...).
Caused by error in stats::cor():
! supply both 'x' and 'y' or a matrix-like 'x'"
In addition: Warning messages:
1: In mclapply(argsList, FUN, mc.preschedule = preschedule, mc.set.seed = set.seed, :
scheduled cores 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 15, 16 did not deliver results, all values of the jobs will be affected
2: In mclapply(argsList, FUN, mc.preschedule = preschedule, mc.set.seed = set.seed, :
scheduled cores 1, 2, 3, 4, 5, 6, 7, 8, 9, 11 did not deliver results, all values of the jobs will be affected

I guess it was caused by the following source code of function calculatePresStats:

if ("cor_kIM" %in% stats) {
        kIM_replicate1 <- kIM_list[[replicate1]]
        kIM_replicate2 <- kIM_list[[replicate2]]
        pres_stats_replicate1_replicate2 <- pres_stats_replicate1_replicate2 %>% 
          dplyr::mutate(cor_kIM = unname(sapply(.data[[id_column]], 
            function(id) {
              stats::cor(kIM_replicate1[[id]], kIM_replicate2[[id]], 
                method = corr_method)
            })))
      }

[anita-termeg]

Hey @LiuCanidk,

First of all, thank you for using our tool and for reporting these issues!

About the first error:
Did you by any chance run this on Windows? What do you get if you run this chunk?

doParallel::registerDoParallel(5)
foreach::getDoParName()

The function vectorize() is defined on line 307 of calculatePresStats.R and should be loaded when you run library(CroCoNet). But whether it’s then passed on to the parallel workers turns out to be dependent on the parallel backend that doParallel selects. I could not reproduce the error on our Linux system, which uses fork-based parallelization. However, if your system uses a PSOCK-like backend, that could explain why the error only surfaces on your end. If this turns out to be the case, I’m happy to implement the necessary changes to pass on the required functions explicitly to the parallel workers.

Small note: vectorize() is not quite the same as as.vector(), so it would be better to stick to the original function.

About the second error:
This might be the result of the warning that you see after the error, which tends to come up if a child process dies mid-computation due to memory issues. All the values corresponding to these processes will be NULL and this might lead to the stats::cor error. I’d recommend reducing the number of cores (e.g. try the original n_cores = 5), or if possible, increasing the allocated memory.

[LiuCanidk]

Hey @LiuCanidk,

First of all, thank you for using our tool and for reporting these issues!

About the first error: Did you by any chance run this on Windows? What do you get if you run this chunk?

doParallel::registerDoParallel(5)
foreach::getDoParName()
The function vectorize() is defined on line 307 of calculatePresStats.R and should be loaded when you run library(CroCoNet). But whether it’s then passed on to the parallel workers turns out to be dependent on the parallel backend that doParallel selects. I could not reproduce the error on our Linux system, which uses fork-based parallelization. However, if your system uses a PSOCK-like backend, that could explain why the error only surfaces on your end. If this turns out to be the case, I’m happy to implement the necessary changes to pass on the required functions explicitly to the parallel workers.

Small note: vectorize() is not quite the same as as.vector(), so it would be better to stick to the original function.

About the second error: This might be the result of the warning that you see after the error, which tends to come up if a child process dies mid-computation due to memory issues. All the values corresponding to these processes will be NULL and this might lead to the stats::cor error. I’d recommend reducing the number of cores (e.g. try the original n_cores = 5), or if possible, increasing the allocated memory.

Thanks for your reply. I followed your suggestion and found no error in Linux when running with n_cores=10 with 30 CPUs. However, I encountered another error when running the following code (just the next step in the tutorial), on the same Linux environment:

random_pres_stats_jk <- calculatePresStats(random_modules, network_list, c("cor_adj", "cor_kIM"), replicate2species, n_cores = 10)

Output:
Calculating cor_adj using weight*direction
Error in { : task 1 failed - "subscript out of bounds"

Error still occur when I change n_cores=1.
Could you please provide any solution about this?

[anita-termeg]

Thanks for confirming that the previous issue is resolved!
For this new issue: since I can’t reproduce it on my end, it’d be really helpful if you could share the exact script that triggered the error. That way I could look into it in more detail.