20th Jul 2008
Django ORM + threading = memleak (workaround)
Well, after trying hundreds of ways to make Python’s carbage collector work with Django’s ORM and threading (see here - scroll to “Python 2.5 bug”) and sing many tools (heapy, valgrind) to try to find the leaks (all the tools show 15-30MB used, no leaks, but in reality program uses all available memory and starts to swap within minutes) I’ve to conclude that there doesn’t seem to be a workaround.
I’ve tried:
- passing only integers instead of Django objects;
- adding +” to strings to make copies of strings, not references, (copy.copy and copy.deepcopy too);
- creating threads inside of a threads, hoping that would lose references somehow;
- del Object; del everything;
- weakrefs;
- moving all Django code into a function and only passing integer to it;
- disabling Django and only leaving lxml (parsing library) in thread, and vice versa - still leaks;
- something else too, but can’t remember all.
The worst part is that nothing detects where those GIGABYTES are going, every tool I used shows 15-30MB memusage.
The workaround I’ve settled for - running separate child processes and connecting to parent “queuer” process via xmlrpc. Takes a lot of memory (each process is 17MB vs some KB for thread) and my guess is that xml isn’t the most effective, but at least no memleaks even if child is running infinite loop.
If you have other ideas to try - let me know.
|