So I was trying to combine a lot of pngs (5k images, 20kb each, 1k by 1k, not that much tbh) into a mpeg/movie and I thought this would take a few hours max.
Oh boy was I wrong on quite a few scales.
First, tried to use ffmpeg to render these matplotlib images in memory and make movie at the same time. After 40h that rendered a SINGLE video out of those images, I decided there has got to be a better way. I chalk this up as Python not being optimized or more likely, me not being smart enough to know what the right thing to do.
Plan B: render those 5K images to disk, then merge post. I TOTALLY thought this cannot be THAT hard and before we get GB of memories, this must be the way people do things in the ancient times like writing flip book animations.
So yeah, looked up ffmpeg, for a series of images not consecutively named, it is annoying to do. Then switched to magick convert. Took a few tried. Maxed out memory (12GB) several times before I realized, probably need disk cache. Have some pretty fancy looking errors like this:
(exception processing is suspended) @ warning/exception.c/ThrowException/1049. convert: unable to extend cache 'input_2018-09-03T120000.png': No space left on device @ error/cache.c/OpenPixelCache/3883.
In the end, took me a few wasted attempt (each 10+h) while I stare at the blank screen before more Google Fu finally lead me to something along the line of this:
magick convert -monitor -define registry:temporary-path=E:/Temp -limit memory 5GiB
I tried too many times to just use last line to combine without thinking the spatial time complexity (thanks, to no compsci background). 2nd line give progress update, 3rd line FORCE a more reasonable temporary space. 4th line prevent RAM overflow. I really hope this is the end of this rendering fiasco. I am not a smart man… 5th line should NOT include combine, just input /output. One liner is good enough I split here for easier readability.
But glad to see the open source community thought about these types of problem through and through: start reading here: https://www.imagemagick.org/script/architecture.php#tera-pixel
Maybe I will link the video rendering later on once it ever finishes. BTW, it takes about 30minutes to write 5K files of 1K x 1K to disk. Sigh. How people use do all these.
Update 1: One liner:
magick convert -monitor -define registry:temporary-path=E:/Temp -limit memory 5GiB *.png movie.mpeg
turned out to be the best. Obviously, update your temporary path and this assume you want to turn everything in the current working directory to the movie. I never expected a simple conversion like this took me almost a few days of on and off GoogleFu to finalize. Hopefully whoever finds this can avoid such hassle.
Convertion time in the order of a few magnitude.