BitBucket SourceTree and AzureDevOps Credential Setup.

This was unnecessarily annoying.

If you have a private GitHub repositories that you would like to hook up to a Git GUI, GitKraken would have been my primary choice but they need pro account for that to happen. However, SourceTree is free for Azure DevOps. BUT, its setup never seems to work somehow, it turned out to be related to bad URL in HTTPS setup.

For detail see this post:

https://community.atlassian.com/t5/Sourcetree-questions/Azure-DevOps-with-Sourcetree-Adding-a-remote-account/qaq-p/1065001

TLDR: use “https://*username*.visualstudio.com” in the address field for BitBucket SourceTree instead of anything like https://dev.azure.com/*username* or whatever long URL you might have copied from Azure Devops portal.

 

Python Flask Image

I have been following this wonderful Flask Mega Tutorial by Miguel Grinberg and really learned a lot about developing and deploying Flask for my work.  Highly recommend for those want to  build simple flask website that scale okay.

What has been bugging me FOREVER about it, was why that particular ancient artifact” image for his tutorial. How is that related to flask? Namely, this:

Powder flask

Image Credit: Left: collections.vam.ac.uk, right Miguel obviously :D.

It turned out to be a 1600 era gun powder flask, as shown here: http://collections.vam.ac.uk/item/O78654/powder-flask-unknown/ and also seems HIGHLY unique one of a kind, as well. I was pointed there through this Pinterest Pin: https://www.pinterest.ca/pin/338684834450893858/

So it makes sense, because we were using Gunicorn to deploy, it needs ammunition (content) which was served via gun powder Flask.

Neat.

Also, this looks so much like AK47 gun magazines, just saying.

 

“selenium.common.exceptions.SessionNotCreatedException: Message: session not created: No matching capabilities found”

I had been banging my head on this error for literally DAYS. Why is Selenium rejecting to launch? I past it some Options but why is it refusing to even launch????

Turned out,

I did this:

from selenium.webdriver.firefox.options import Options

and tried to past it to a Chrome WebDriver as “Options” when I should have imported

from selenium.webdriver.chrome.options import Options

I am not a smart man…

I was googling so much and the top Google results mentioned using OperaDrivers, using Edge Drivers etc and running into similar issues. Most likely, some sort of configurations are past wrong. In my case, it was very obvious. I even went digging into ChromeDriver vs Chrome version compatibility issue (just leave TravisCI to ensure their compatibility but print out versions explicitly doesn’t hurt… either)

 

Quick shallow insight into ML related workflow management packages as of 2019-11-11T204512EST

This is a very early draft of the information I have gathered all over the internet. I am interested and are in the process of checking the highlighted features. However, so far, MLFlow seems to be winning out and its interest are actually on PAR with KubeFlow (given how KubeFlow is much more… general purpose, this seems like a massive surprise).

ScreenShot_2019-11-11T204433EST.png

Interestingly, outside of MLFlow, the other competitiors seems to be really having a PR issue. I had to exclude Spell and Pachyderm because of word commoness. Therefore, I will most likely be exploring MLFlow to hope gain some expertise with it in the upcoming days.

ScreenShot_2019-11-11T205003EST.png

 

 

Annoying MLFlow Artifact Servicing Tutorial Windows Bug

So I was following the otherwise excellent MLFlow official tutorial on Windows and keep running into this annoying bug while attempting to serve the model:

mlflow.exceptions.MlflowException: Could not find a registered artifact repository for: c:/XXXX/XXXXX/data_mlflow/mlflow/examples/sklearn_elasticnet_wine/mlruns/0/
b02a6befa2824c339891b088803b723e/artifacts/model. Currently registered schemes are: [”, ‘file’, ‘s3’, ‘gs’, ‘wasbs’, ‘ftp’, ‘sftp’, ‘dbfs’, ‘hdfs’, ‘viewfs’, ‘runs’, ‘models’]”

I was like… WTF? I am following the official tutorial word by word without any typing and how could it brick???

It turned out to be an issue with MLFlow not handling path very well on Windows (probably should have used pathlib?

The issue as pointed in the GitHub issue here: https://github.com/mlflow/mlflow/issues/1193#issuecomment-490418110

As pointed out by enpinzolas, you should replace prefix “C:/xxxx” in the path string with “file://C:/xxxx”.

This fixed it for me and I was able to serve the training example properly.

In hindsight, the error message makes sense. “C:” was not in that list given. “file:” is. They REALLY should have better parsed that link.  Most likely because the nature of the adaptability. A small PR should be in order to fix this.

 

Post Mortem of Kaggle Servestal Steel Defect Competition

That was an even more epic fail compare to the weather data competition host seven months back.

I am about 19 hours away from the deadline but with me tomorrow full time at this deep learning conference, I know this is not going well so I might as well get a head start on this post mortem.

Yet again, I failed to submit to Kaggle competition. Fool me once, shame on me. Fool me twice… damn. I am a fucking idiot.

Things I should have done better:

  • I completed all my training locally on my own GPUs for accessibility reasons. Also I organized my data/folders/dependency scripts in my own particular fashion (not yet as a python module package published to pip… hence also not readily callable from Jupyter)
  • Import modules were not robust enough and not working as they should.
  • /kaggle/work, /kaggle/input folders etc are completely foreign concept to me. I organized data my own way… very very differently.
  • While prediction was working (but not really visually checked), submission to Kaggle was not. The RLE encoding and decoding schema was notoriously buggy, annoying to work with and difficult to bug check. This lead to almost 75% of the dev time preparing for these. Still, it has bug in the end at even the local validation stage. Next time, i will just use a package or some tried and true public code to do this. God damn it.
  • Speaking of dev time, 3 hours per week was already far cry from enough. But having 2 full weeks of almost 100% off hours dedicated to ML interview preparation was TERRIBLE idea. Oh, also I was away on vacation for one full week too. These all happened starting one month before the Kaggle deadline are very very bad combinations. Need to clear calendar for the two weeks BEFORE Kaggle deadlines.
  • Overall, I think all these epic fails are the direct results of invalidated/insufficient data pipeline works and lack of iterative improvement submission processes. The deep learning part was not too bad (as in, it at least runs, but no clue how well it generalized) but the the check/iterate/improve portion was very very poorly handled. Despite the recent post interview data pipeline improvement, it failed to integrate with the Kaggle Kernel computation environment.
  • More importantly, the lack of offline integration with online Jupyter analyses pipeline was also likely the cause of multiple analyses bug issues.

Tomorrow I am at the 2019 REWork DeepLearning summit and I will see what I can fix but given that my DeepLabV3+ models are not output binary labels, these are some very terrible signs. Also, from teh 836+ test images masks CSV generated, having two pixels of defects suggests… terrible performance. Shoudl have examined these sooner. Not the day of the submission… sigh.

 

I peaked in terms of programming performance

I was working on this Flask app, which require data base commitment after data model changes.

I wrote:

db.seesion.coomit()

In a SINGLE line of code composed of THREE English words, I manage to make two consecutive typos, took two debug sessions to discover the bugs. Impressive indeed.

This is the reason I do not work on production system and never touch anything mission critical.

 

Ignorance is not bliss

Occasionally, I help others build website.s

One client of mine approached me asking me to help them out. Sure thing, no issues. You want a website, you get a website. Nothing fancy.

Then I emailed to ask what domain name they wanted and then they turned back asking me what is a “domain name”, is that the name of the website? I hesitated, pondering my response, purpose of my existence, and how to best respond to that question.

I could:

  1. LMGTFY: passive, but aggressive and maybe teach a lesson?
  2. Explain the concept: teach a lesson?
  3. Pretend to never seen this email?
  4. Drop the client and move on with my life?
  5. Teach them how to look up an unknown concept? Because surely, everyone in the world has heard Wikipedia by now?

It really takes less than 5 minutes to educate yourself to learn something new.

Not everyone does that.

I will try to stick with those who at least attempts to educate themselves.

But I have to work with people like that. In the end, I wrote her a short email explaining that.

Such, is the fate of working with people. You do what needs to be done to get things moving.

 

That moment when you realize you made a mistake… is exhilarating.

Upgraded my old laptop SSD to NVME 1TB recently via cloning through a external M.2 enclosure. All went well. Decided to recycle the old laptop SSD using that enclosure for a SPEEDY USB.

Massive giddy ensues.

Then realize… can never see it format on Win/Linux fail to even initialize. Much research later, it dawned upon me that old drive is SATA M2 SSD with B+M keys and new enclosure only take M keys NVME M.2.

Now back to order B+M keys compatible SSD enclosure while my M NVME enclosure sits empty, yearning for speedy copying…