That was an even more epic fail compare to the weather data competition host seven months back.
I am about 19 hours away from the deadline but with me tomorrow full time at this deep learning conference, I know this is not going well so I might as well get a head start on this post mortem.
Yet again, I failed to submit to Kaggle competition. Fool me once, shame on me. Fool me twice… damn. I am a fucking idiot.
Things I should have done better:
- I completed all my training locally on my own GPUs for accessibility reasons. Also I organized my data/folders/dependency scripts in my own particular fashion (not yet as a python module package published to pip… hence also not readily callable from Jupyter)
- Import modules were not robust enough and not working as they should.
- /kaggle/work, /kaggle/input folders etc are completely foreign concept to me. I organized data my own way… very very differently.
- While prediction was working (but not really visually checked), submission to Kaggle was not. The RLE encoding and decoding schema was notoriously buggy, annoying to work with and difficult to bug check. This lead to almost 75% of the dev time preparing for these. Still, it has bug in the end at even the local validation stage. Next time, i will just use a package or some tried and true public code to do this. God damn it.
- Speaking of dev time, 3 hours per week was already far cry from enough. But having 2 full weeks of almost 100% off hours dedicated to ML interview preparation was TERRIBLE idea. Oh, also I was away on vacation for one full week too. These all happened starting one month before the Kaggle deadline are very very bad combinations. Need to clear calendar for the two weeks BEFORE Kaggle deadlines.
- Overall, I think all these epic fails are the direct results of invalidated/insufficient data pipeline works and lack of iterative improvement submission processes. The deep learning part was not too bad (as in, it at least runs, but no clue how well it generalized) but the the check/iterate/improve portion was very very poorly handled. Despite the recent post interview data pipeline improvement, it failed to integrate with the Kaggle Kernel computation environment.
- More importantly, the lack of offline integration with online Jupyter analyses pipeline was also likely the cause of multiple analyses bug issues.
Tomorrow I am at the 2019 REWork DeepLearning summit and I will see what I can fix but given that my DeepLabV3+ models are not output binary labels, these are some very terrible signs. Also, from teh 836+ test images masks CSV generated, having two pixels of defects suggests… terrible performance. Shoudl have examined these sooner. Not the day of the submission… sigh.