Post Mortem of Kaggle Servestal Steel Defect Competition

That was an even more epic fail compare to the weather data competition host seven months back.

I am about 19 hours away from the deadline but with me tomorrow full time at this deep learning conference, I know this is not going well so I might as well get a head start on this post mortem.

Yet again, I failed to submit to Kaggle competition. Fool me once, shame on me. Fool me twice… damn. I am a fucking idiot.

Things I should have done better:

  • I completed all my training locally on my own GPUs for accessibility reasons. Also I organized my data/folders/dependency scripts in my own particular fashion (not yet as a python module package published to pip… hence also not readily callable from Jupyter)
  • Import modules were not robust enough and not working as they should.
  • /kaggle/work, /kaggle/input folders etc are completely foreign concept to me. I organized data my own way… very very differently.
  • While prediction was working (but not really visually checked), submission to Kaggle was not. The RLE encoding and decoding schema was notoriously buggy, annoying to work with and difficult to bug check. This lead to almost 75% of the dev time preparing for these. Still, it has bug in the end at even the local validation stage. Next time, i will just use a package or some tried and true public code to do this. God damn it.
  • Speaking of dev time, 3 hours per week was already far cry from enough. But having 2 full weeks of almost 100% off hours dedicated to ML interview preparation was TERRIBLE idea. Oh, also I was away on vacation for one full week too. These all happened starting one month before the Kaggle deadline are very very bad combinations. Need to clear calendar for the two weeks BEFORE Kaggle deadlines.
  • Overall, I think all these epic fails are the direct results of invalidated/insufficient data pipeline works and lack of iterative improvement submission processes. The deep learning part was not too bad (as in, it at least runs, but no clue how well it generalized) but the the check/iterate/improve portion was very very poorly handled. Despite the recent post interview data pipeline improvement, it failed to integrate with the Kaggle Kernel computation environment.
  • More importantly, the lack of offline integration with online Jupyter analyses pipeline was also likely the cause of multiple analyses bug issues.

Tomorrow I am at the 2019 REWork DeepLearning summit and I will see what I can fix but given that my DeepLabV3+ models are not output binary labels, these are some very terrible signs. Also, from teh 836+ test images masks CSV generated, having two pixels of defects suggests… terrible performance. Shoudl have examined these sooner. Not the day of the submission… sigh.

 

I peaked in terms of programming performance

I was working on this Flask app, which require data base commitment after data model changes.

I wrote:

db.seesion.coomit()

In a SINGLE line of code composed of THREE English words, I manage to make two consecutive typos, took two debug sessions to discover the bugs. Impressive indeed.

This is the reason I do not work on production system and never touch anything mission critical.

 

Ignorance is not bliss

Occasionally, I help others build website.s

One client of mine approached me asking me to help them out. Sure thing, no issues. You want a website, you get a website. Nothing fancy.

Then I emailed to ask what domain name they wanted and then they turned back asking me what is a “domain name”, is that the name of the website? I hesitated, pondering my response, purpose of my existence, and how to best respond to that question.

I could:

  1. LMGTFY: passive, but aggressive and maybe teach a lesson?
  2. Explain the concept: teach a lesson?
  3. Pretend to never seen this email?
  4. Drop the client and move on with my life?
  5. Teach them how to look up an unknown concept? Because surely, everyone in the world has heard Wikipedia by now?

It really takes less than 5 minutes to educate yourself to learn something new.

Not everyone does that.

I will try to stick with those who at least attempts to educate themselves.

But I have to work with people like that. In the end, I wrote her a short email explaining that.

Such, is the fate of working with people. You do what needs to be done to get things moving.

 

That moment when you realize you made a mistake… is exhilarating.

Upgraded my old laptop SSD to NVME 1TB recently via cloning through a external M.2 enclosure. All went well. Decided to recycle the old laptop SSD using that enclosure for a SPEEDY USB.

Massive giddy ensues.

Then realize… can never see it format on Win/Linux fail to even initialize. Much research later, it dawned upon me that old drive is SATA M2 SSD with B+M keys and new enclosure only take M keys NVME M.2.

Now back to order B+M keys compatible SSD enclosure while my M NVME enclosure sits empty, yearning for speedy copying…


 

Fix for Corsair Effects Engine (CEE) for K70 MKII Keyboard Non-Detection Issues [Only Tested on k70 MKII RapidFire RGB]

TLDR: 1) Find your PID. 2) Find Devices.xml of the app and update the PID values for your device in the xml.
Corsair Effects Engine is a super cool utility by Emily Maxwell that makes Corsair keyboard has Spectrograph :). Came across this annoying issue preventing my K70 MK2 RapidFire RGB from being detected by CEE. Endless Google/Reddit ensued suggest MKIIs were not supported. A bit investigation turned out to be a relatively simple PID mismatch issue resulted in keyboard not being detected. So far, it seems to have worked well for me and have not encountered any terrible bugs yet.
To find your own PID device (See graphs below)
  1. Go to “Control Panel\Hardware and Sound\Devices and Printers
  2. Right click your keyboard then properties.
  3. Click Hardware Tab.
  4. Select HID Keyboard Device
  5. Click on Properties
  6. Go to Details Tab.
  7. Select Hardware IDs.
  8. Find the number that shows after PID_: my keyboard is 1B49.

2019-09-29T000920_CORSAIR K70 RGB MK.2 Mechanical Gaming Keyboard Properties (1).png
Steps to Find your PID

 

Find and Update “Devices.xml” (See graphs below) :
  1. Open CEE (not minimized, so it shows under Task Manager – Apps)
  2. Right click “Corsair Effeects Engine” in Process manager, then Open File Location)
  3. Find the “Devices.xml” in that folder, for me it was something ridiculous like : “C:\Users\{USERNAME}\AppData\Local\Apps\2.0\LVJMT5M2.EWQ\T71V1VBW.L83\cors..tion_0000000000000000_0000.0003_9cd37b466552489d”. Yours is very likely to be different too. This is determined by the installer of CEE.
  4. Use Notepad or notepad++ to open the Devices.xml file.
  5. Scroll through and find the device most similar to yours. I suspect more of them should work fine once PID updated. For me, I suspect most K70 devices are MK1 instead of MK2 so I chose “K70 RAPIDFIRE RGB”
  6. Update the PID key values between <PID></PID> for your device. For me, it WAS “0x1B38” and I changed it to “0x1B49” found above and it now works.
  7. Restart the CEE.
  8. Enjoy your RGB Spectrograph if that has been missing in your life?

2019-09-29T002009_Image.png
Finding Devices.xml Location

2019-09-29T001209_Notepad.png
Where to update the PID. Most likely just needs to change the last few digits.

Finding Devices.xml locationWhere to update your PID (just change the last two digits is usually fine. Only tested on K70 RAPIDFIRE RGB but given that all of them use similar matrix, it should work for other K70 as well.)


 

Sigh…

So today a LinkedIn tech recruiter (who also happen to be a nice lady) and I got on the phone. After initial nice greeting, I managed to fumbled my reply to be: “So, what I can do to you today…” instead of the more normal human interaction like “what can I do FOR you today?” I am so getting sued. LOL.

 

Just want to get this out there… Time is precious and it accumulates well when saved properly.

I commute 15 minutes from home to work by car (yes, very lucky). I rarely bothered to pair my Bluetooth with the car audio to listen to anything productive and just usually let radio play or silence.

Then, I started forcing myself to take the extra few minutes to hook up the bluetooth to the car (not same car, car sharing) and listen to some audible book, before you realize it, I finished several audio books already. 15 minutes per trip, 30 minutes per day, 3.5 hours per week and two weeks of commuting gets 7 hours of audio content which is typically the length of a shorter book. Get some exciting book like those that teach you to negotiate better, or tell you about cool history or something. I am glad Audible is making me the habbit of listening to stuff again. I am very grateful.


 

System.Runtime.InteropServices.ExternalException : A generic error occurred in GDI+.

This little sucker took me wayyyyyyyyyyyy too long to figure out. I was coding in C# on Windows and this error popped up when trying to save a file.

What it probably should have said, is “Dude, I cannot save your shxt to a file. Something went wrong with file creation process that before I write data to it, it stopped working. Find that out, fix it!”

I was trying to save an EMGU image to a path but keep getting this error. It turned out how I have specified the file path incorrectly to the tune of “AAAA\BBBB\CCC\.jpeg” and that file creation process was not valid unless with a properly named file (without . at the beginning most likely)