Yesterday we succeeded in getting CSV's to generate from TMPO live on the website and send them to the browser. We however noticed that each request uses some memory and fails to free it afterwards. After a few requests the server unavoidably crashes.
We have tried following things to reduce memory load and free it up after the request, however none have really worked.
- Write a wrapper to close the file buffer after the request has completed. (link)
- Use a temporary file to store the CSV and serve it instead of using StringIO or cStringIO.
- Setting the Flask flag
app.use_x_sendfile = True, to have nginx serve the file directly instead of the app. (I did not thoroughly test this, not sure of its effect)
- Deleting the Pandas DataFrame after the CSV is written, using
del df
- Calling the garbage collector after the delete:
import gc; gc.collect() (link)
Does anybody have other ideas we could try? The download page is live, but hidden under opengrid.be/download. The status quo is that it does work, however after a few runs it will crash the server, which then immediately restarts.
Yesterday we succeeded in getting CSV's to generate from TMPO live on the website and send them to the browser. We however noticed that each request uses some memory and fails to free it afterwards. After a few requests the server unavoidably crashes.
We have tried following things to reduce memory load and free it up after the request, however none have really worked.
app.use_x_sendfile = True, to have nginx serve the file directly instead of the app. (I did not thoroughly test this, not sure of its effect)del dfimport gc; gc.collect()(link)Does anybody have other ideas we could try? The download page is live, but hidden under opengrid.be/download. The status quo is that it does work, however after a few runs it will crash the server, which then immediately restarts.