Azure Function x64 Gotcha

This was super annoying when I ran into it.

I was playing with Azure function for a bit and because our app needs to be 64 bits for dependency reason, I made a simple Azure Function v2 to apply the serverless technology for this situation.

However, what was working as a straight copy and paste working app now stopps working somehow. For no reasons at all.

All I did was changing the build to x64 from x86 and it stopped and died with errors similar to this:

[2019-12-22 19:30:04] A host error has occurred during startup operation '2236efae-de7b-4b80-9569-a009ede8c172'.
[2019-12-22 19:30:04] System.Private.CoreLib: Could not load file or assembly 'AzureIntegration, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null'.

With NO OTHER details whatsoever.

It turned out, by default Visual Studio 2019 create 32 bit Azure Function projects and if I change it to x64 bits, the AzureFunction CLI for local debugging needs to be x64 bit as well. So, follow this guide fixed the problem:

https://github.com/Azure/azure-functions-core-tools/wiki/Running-64bit-functions-in-Visual-Studio

Basicly, download the x64 release of the azure function core tool and then set “Debug” to launch executable, pointing to the func.exe from the download and then add application arguments of “host start” and then it is all launching fine!

Really hope this will not be an issue in the future.

Thanks to Henk Boelman’s blog entry to point me to start at least investigating the x64 as the beginning of this issue: https://www.henkboelman.com/articles/azure-function-x64-dev-setup/

 

Pro Tip for Easier Time Reconsiling/Accounting Transactions

I might have posted this before, but if you ever keep track of your own personal finance using software/excel/YNAB/Mint/whatever, especially when it comes to transfer between accounts (does not , it is always better than use weird specific amount than whole amount.

For instance, if you need to transfer $500 fom to your life of credit to bank account, try to transfer 521.12 or some non-integer numbers so the chance of this transaction being unique is much higher. This way, you will never have to worry about mixing transactions. Avoid 100, 1000, 500 etc amount as once the number of accounts and transactions scale over time, it can be super confusion and lead to much later issues when reconsiling whether that 100$ was the one you sent to Joe or the one you transferred to your saving or the 100$ that you were putting towards your retirement.

Most importantly, it makes subsequent search so much easier.

 

BitBucket SourceTree and AzureDevOps Credential Setup.

This was unnecessarily annoying.

If you have a private GitHub repositories that you would like to hook up to a Git GUI, GitKraken would have been my primary choice but they need pro account for that to happen. However, SourceTree is free for Azure DevOps. BUT, its setup never seems to work somehow, it turned out to be related to bad URL in HTTPS setup.

For detail see this post:

https://community.atlassian.com/t5/Sourcetree-questions/Azure-DevOps-with-Sourcetree-Adding-a-remote-account/qaq-p/1065001

TLDR: use “https://*username*.visualstudio.com” in the address field for BitBucket SourceTree instead of anything like https://dev.azure.com/*username* or whatever long URL you might have copied from Azure Devops portal.

 

Python Flask Image

I have been following this wonderful Flask Mega Tutorial by Miguel Grinberg and really learned a lot about developing and deploying Flask for my work.  Highly recommend for those want to  build simple flask website that scale okay.

What has been bugging me FOREVER about it, was why that particular ancient artifact” image for his tutorial. How is that related to flask? Namely, this:

Powder flask

Image Credit: Left: collections.vam.ac.uk, right Miguel obviously :D.

It turned out to be a 1600 era gun powder flask, as shown here: http://collections.vam.ac.uk/item/O78654/powder-flask-unknown/ and also seems HIGHLY unique one of a kind, as well. I was pointed there through this Pinterest Pin: https://www.pinterest.ca/pin/338684834450893858/

So it makes sense, because we were using Gunicorn to deploy, it needs ammunition (content) which was served via gun powder Flask.

Neat.

Also, this looks so much like AK47 gun magazines, just saying.

 

“selenium.common.exceptions.SessionNotCreatedException: Message: session not created: No matching capabilities found”

I had been banging my head on this error for literally DAYS. Why is Selenium rejecting to launch? I past it some Options but why is it refusing to even launch????

Turned out,

I did this:

from selenium.webdriver.firefox.options import Options

and tried to past it to a Chrome WebDriver as “Options” when I should have imported

from selenium.webdriver.chrome.options import Options

I am not a smart man…

I was googling so much and the top Google results mentioned using OperaDrivers, using Edge Drivers etc and running into similar issues. Most likely, some sort of configurations are past wrong. In my case, it was very obvious. I even went digging into ChromeDriver vs Chrome version compatibility issue (just leave TravisCI to ensure their compatibility but print out versions explicitly doesn’t hurt… either)

 

Quick shallow insight into ML related workflow management packages as of 2019-11-11T204512EST

This is a very early draft of the information I have gathered all over the internet. I am interested and are in the process of checking the highlighted features. However, so far, MLFlow seems to be winning out and its interest are actually on PAR with KubeFlow (given how KubeFlow is much more… general purpose, this seems like a massive surprise).

ScreenShot_2019-11-11T204433EST.png

Interestingly, outside of MLFlow, the other competitiors seems to be really having a PR issue. I had to exclude Spell and Pachyderm because of word commoness. Therefore, I will most likely be exploring MLFlow to hope gain some expertise with it in the upcoming days.

ScreenShot_2019-11-11T205003EST.png

 

 

Annoying MLFlow Artifact Servicing Tutorial Windows Bug

So I was following the otherwise excellent MLFlow official tutorial on Windows and keep running into this annoying bug while attempting to serve the model:

mlflow.exceptions.MlflowException: Could not find a registered artifact repository for: c:/XXXX/XXXXX/data_mlflow/mlflow/examples/sklearn_elasticnet_wine/mlruns/0/
b02a6befa2824c339891b088803b723e/artifacts/model. Currently registered schemes are: [”, ‘file’, ‘s3’, ‘gs’, ‘wasbs’, ‘ftp’, ‘sftp’, ‘dbfs’, ‘hdfs’, ‘viewfs’, ‘runs’, ‘models’]”

I was like… WTF? I am following the official tutorial word by word without any typing and how could it brick???

It turned out to be an issue with MLFlow not handling path very well on Windows (probably should have used pathlib?

The issue as pointed in the GitHub issue here: https://github.com/mlflow/mlflow/issues/1193#issuecomment-490418110

As pointed out by enpinzolas, you should replace prefix “C:/xxxx” in the path string with “file://C:/xxxx”.

This fixed it for me and I was able to serve the training example properly.

In hindsight, the error message makes sense. “C:” was not in that list given. “file:” is. They REALLY should have better parsed that link.  Most likely because the nature of the adaptability. A small PR should be in order to fix this.

 

Post Mortem of Kaggle Servestal Steel Defect Competition

That was an even more epic fail compare to the weather data competition host seven months back.

I am about 19 hours away from the deadline but with me tomorrow full time at this deep learning conference, I know this is not going well so I might as well get a head start on this post mortem.

Yet again, I failed to submit to Kaggle competition. Fool me once, shame on me. Fool me twice… damn. I am a fucking idiot.

Things I should have done better:

  • I completed all my training locally on my own GPUs for accessibility reasons. Also I organized my data/folders/dependency scripts in my own particular fashion (not yet as a python module package published to pip… hence also not readily callable from Jupyter)
  • Import modules were not robust enough and not working as they should.
  • /kaggle/work, /kaggle/input folders etc are completely foreign concept to me. I organized data my own way… very very differently.
  • While prediction was working (but not really visually checked), submission to Kaggle was not. The RLE encoding and decoding schema was notoriously buggy, annoying to work with and difficult to bug check. This lead to almost 75% of the dev time preparing for these. Still, it has bug in the end at even the local validation stage. Next time, i will just use a package or some tried and true public code to do this. God damn it.
  • Speaking of dev time, 3 hours per week was already far cry from enough. But having 2 full weeks of almost 100% off hours dedicated to ML interview preparation was TERRIBLE idea. Oh, also I was away on vacation for one full week too. These all happened starting one month before the Kaggle deadline are very very bad combinations. Need to clear calendar for the two weeks BEFORE Kaggle deadlines.
  • Overall, I think all these epic fails are the direct results of invalidated/insufficient data pipeline works and lack of iterative improvement submission processes. The deep learning part was not too bad (as in, it at least runs, but no clue how well it generalized) but the the check/iterate/improve portion was very very poorly handled. Despite the recent post interview data pipeline improvement, it failed to integrate with the Kaggle Kernel computation environment.
  • More importantly, the lack of offline integration with online Jupyter analyses pipeline was also likely the cause of multiple analyses bug issues.

Tomorrow I am at the 2019 REWork DeepLearning summit and I will see what I can fix but given that my DeepLabV3+ models are not output binary labels, these are some very terrible signs. Also, from teh 836+ test images masks CSV generated, having two pixels of defects suggests… terrible performance. Shoudl have examined these sooner. Not the day of the submission… sigh.

 

I peaked in terms of programming performance

I was working on this Flask app, which require data base commitment after data model changes.

I wrote:

db.seesion.coomit()

In a SINGLE line of code composed of THREE English words, I manage to make two consecutive typos, took two debug sessions to discover the bugs. Impressive indeed.

This is the reason I do not work on production system and never touch anything mission critical.