Welcome to My Blog

Freaking… OSX and its subtle security feature

So… Be me. Tried to install VirtualBox on OSX High Sierra. Always fails. But it is installed. When run it, it says kernel and other stuff are not even installed. Wondering how this can happen.

So furious google ensured.

Then found this stackexchange thread… on this issue… and the causes are more bizarre.

Please review the following two posts to see the excellent problem descriptions:

TLDR: apparently, OSX DELIBERATELY HIDES the “Allow” button (or Windows UAC equivalent? XD) WHEN there is a mouse sharing program running PREVENTING you from allowing installation and elevate permissions etc. Neat security bulletproofing but at the cost of user convenience with ZERO warning WHATSOEVER. I had ShareMouse running and maybe even TeamViewer, closed both of them and the “Allow” from the Security and Privacy section worked and VirtualBox installed fine. What.The.Hell…

 
Posted in Uncategorized

Updating PIP in Windows

Ran into this enough times that it frustrates me enough to write this down:

When running python from old anaconda distros on Windows and need to update pip, don’t use pip to update pip because Windows don’t let that happen. So instead:

  1. Launch Admin Anaconda Prompt
  2. Run: python -m pip install –upgrade pip

 

 
Posted in Uncategorized

DataLad… on Windows

Hmm… that was surprisingly painful to get it all working.

  1. Must ensure python is installed. admin permission run pip update and install all dependencies seems to have helped. Even though I started with Anaconda, that took longer than expected.
  2. “pip install datalad”
  3. install Git (latest 32bit as recommended by Git-Annex): be careful here as the path is actually best done by confining it strictly to Git Console (so not to mess up your other function).
  4. Install Git-Annex from their website (this is NOT installed by DataLad, but required).
  5. Launch Git Console.
  6. Test Git
  7. Test Git Annex.
  8. … and it seems datalad works from that console, only.

I did still ran into bugs like and I failed to see anything being majorly changed in the folder I was running this from: .datalad, .git etc:

$ datalad save . -m "Initial commit of all T2 data from CNBP registry before any kind of conversion"
Failed to run ['git', '-c', 'receive.autogc=0', '-c', 'gc.auto=0', '-c', 'core.bare=False', '--work-tree=.', 'annex', 'proxy', '--', 'git', 'add', '--update', '--verbose'] under 'H:\\DataLadExp'. Exit code=1. out=
failed
err=git-annex: .git\annex\misctmp\proxy.0: removeDirectoryRecursive:removeContentsRecursive:RemoveDirectory "\\\\?\\H:\\DataLadExp\\.git\\annex\\misctmp\\proxy.0": unsatisfied constraints (The directory is not empty.)
git-annex: proxy: 1 failed

failed
git-annex: .git\annex\misctmp\proxy.0: removeDirectoryRecursive:removeContentsRecursive:RemoveDirectory "\\\\?\\H:\\DataLadExp\\.git\\annex\\misctmp\\proxy.0": unsatisfied constraints (The directory is not empty.)
git-annex: proxy: 1 failed

So I guess it is not entirely working yet…

 

Update: on second try with SHORTER commit message it seems to worked. Not sure if it is a folder creation/permission bug but… it “save(okay) wow

Update1: Just tried it on another computer…  and it is complaining Datalad is missing from the git-bash window. Hmmmm……I am definitely doing something wrong here… The mystery deepens.

Update 2: Finally figured out that C:\ProgramData\Anaconda\Scripts must be added to the Environment search path for Git Bash to recognize the exe in the search path to run datalad executable without issue. So to recap: 1) Must be from Git-Bash windows because only it can reference Git/Git-Annex properly. 2) Path must include to executable DataLad. Once all these are set, datalad SHOULD WORK in the Git-Bash window. 3) Test show that Anaconda Console/Terminal/Bash will not work as it will not recognize Git/Git-Annex properly… unless you somehow configured that properly? I swear this is how DLL hell used to start…

 
Posted in Uncategorized

fnmatch and its OS specific behaviour

Another funny quicks about Python fnmatch algorithm.

I was playing around with Pydicom, and it has a function called: get_testdata_files which calls get_files from its data_manager.py which calls fnmatch from fnmatch.py which is part of Anaconda 3 lib. However, fnmatch itself is quicky in that it will enforce case matching on Linux but relax this on Windows.  So When calling the Pydicom function with “JPEG” will only match fewer test file on ubuntu (e.g. Travis-CI) vs when testing this on Windows (my dev machine).

Use fnmatchcase are probably a better idea.

 
Posted in Uncategorized

Emgu 3 and its fun quirks

Just spent a day finding out why Emgu seems to give me one hell of a time computing simple pca for a bunch of 2d points.

For some reason we are not sure and yet replicated across two dev machines, if you pass wrong matrix size as output parameters to functions like CVinvoke.calcCoVariance or CVinvoke.eigen it will crash the program but in very insidious ways with no error given except the output matrices just contain data of all 0s. And left you wondering what the hell went wrong….

Also calccovariance’s mandatory flag are apparently require “or” somehow in our layout such as “Emgu.CV.CvEnum.CovarMethod.Normal | Emgu.CV.CvEnum.CovarMethod.Cols” worked for us. Then afterwards the covariance matrix needs to normalize by array length size afterward…. In order to get the same results as numpy

Lastly, this one is on me: eigen value and vectors are arranged based on descending order in emgucv/opencv.

Almost have my pca working as intended to identify principle two dimensional clusters distributions…

 
Posted in Uncategorized

YNAB and its annoyance

Words cannot express the amount of joy when I first started using YNAB. It felt leaps and decade better than old Mint which has not really been updated since a decade ago?

So… then YNAB TD Canada Trust importer broke around Feb/March of this year, not YNAB’s fault, I know. But the fact it was not able to connect my bank account for like three months while MINT can still do 3 days later after TD Canada interface update makes me wonder if YNAB should switch to a better bank connector?

That part… is not the worst.

So today after everything is finally back up (btw, I do have to pay for YNAB, and not Mint), three months later. I was happily synchronizing everything. When I realized that the bank balance are not correct... this is where shxt really hits the fans. As the core principle and competency of a finance/accounting software, the number one creed of its goal above and beyond everything is the get its shit together and get transactions imported/transferred correctly from the bank to the BEST of its abilities. Turned out YNAB only import ONE month back and even back then, some dates were offsetted by as far as 4 days.

If YNAB requires ME to manually cross validate every single transactions to rebalance the book vs bank record, we have a major problem here as not only is it making my life more difficult vs Mint, it is now forcing me to run a full investigation to see WHICH lines are not imported/date set properly! So far, rough comparisons suggest everything in March is pretty much gone.

A few quick comparisons: Note blue transactions are only there because of a LINK from those credit cards etc.

2018-05-21_T111706_wpc4szpr37nruck6domhlk7b9.png

I will update again once I begin to sort through the mess.

Update 1: spent past 24h cleaning up the update. A few piece of good news: manual import of QBX format has auto deduplication built into it (at least for my main banking account) so that really helped out. I had to blank import past 6 months of statement to patch up the missing bits. My suspicion is that when missing import sometimes, the transactions gets neglected and will no longer be imported. i.e. completely fall off the book. =_=…. While doing this for another different account, I actually noticed that the deduplication process fails so resulted in crap tons of duplicated entries and causing issues…. again, more headaches. This requirement to manually hunt down when and how the out of sync occured really leaves terrible taste in my mouth as it fails at a basic level as I mentioned earlier: 1) if this software is PULLING data from the bank in the first place after I give it permission, it should not be requiring me to go over every single transactions again to en sure that they are synced over, that is not the job of the user, that is the job of the software. As a user of financial software, my goal and sole purpose of using YNAB or mint or the like is to do budgeting, tracking expenses, not questioning the authenticity of whether the copy from the bank over to the software occurred. If I want to do that, I could have used a spreadsheet…/rant.

 
Posted in Uncategorized

Plotly and Google Colab

Plotly is super nice as it fixed the critical issue of non-interactivity in Python MatPlotLib. Sure MatPlotLib is superpowerful but still, you need to iterate likemad if you want to change anything and is not super easy to explore data points on mouseover etc. Plotly is quite nice but does not play well with CoLab environment unless you call initialization in EVERY SINGLE CELL. Luckily, someone solved that problem for you. Check out this StackExchange answer:

https://stackoverflow.com/questions/47230817/plotly-notebook-mode-with-google-colaboratory

Yeah, another weird quirk of Google Colab vs regular jupyter notebook…

 

 
Posted in Uncategorized

CentOS, phpMyAdmin Fun

That actually turned out not TOO too painful but still annoying.

I hate doing anything web if the error is so ambigious… like… access denied… but WHY? FUCKING WHY??????

So followed this article:

https://www.liquidweb.com/kb/how-to-install-and-configure-phpmyadmin-on-centos-7/

Not too bad. Got it working.

However, then had to make sure apache has access to all of the phpmyadmin so had to chown for all that.

chown -R apache:apache somethingsomething like that.

Then had to change the permission in phpmyadmin.conf to ensure permission is open.

Then because it is behind proxy, you got add this: https://coderwall.com/p/ev5mza/solve-phpmyadmin-path-problems-behind-apache-proxy

Lastly, I still did not configure it properly and had to call the index.php instead. =_=

At least, 3 hours of adventure later, it is somewhat working. Adding more stuff ot the security audit list.

 

 

Also something I learned along the fun. sudo -i to enter directory that I don’t have permission but if I have sudo to enter… sigh. i probably broken like a gazillion security rules etc but hey, whatever get the project moving forward…

 

 
Posted in Uncategorized

CentOS7 and Docker

So… apparently, Docker service needs to be started: sudo service docker start… before you can pull any Docker images or else you get “Warning: failed to get default registry endpoint from daemon (Cannon connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?)” error. Hey, for once… I guess the error was quite helpful and on point and less ambiguous. Probably should figure out a way to auto run the service. Too bad I am still so new to CentOS. SELinux 0. lol

 
Posted in Uncategorized

HP z620 and Fancy GPUs

So. Had to upgrade a HP Z620 workstation at work lately to get a 1070 board to fit to do some ML work soonish.

A few things I noticed because I was not the only one googling like mad about it:

1) does it have enough power? 2) does it have enough room 3) what cable is required?

  1. Yes, the native power supply has like almost 700W so more than enough for puny 1070 power requirement which is less than 200W.
  2. BARELY. I originally had the old Quadro card further up and had to trade spot with the 1070. Keep in mind you will need to remove a few things.  Move the original GPU to lower spot or else the 1070 literally won’t have enough breathing room (as in, it will try to suck air from the bottom metal plate which… will most likely cause issues and prevent heat dissipation (unless you use the case to vent heat via conduction? which would be kind of cool yet hot I guess? (Don’t use the black slot shown here for 1070 or any recent large size card…). YOU MUST REMOVE a metal spacer from the chassis cover in order to fit 1070 though as the card is TOO wide. Luckily, Z620 had designed the chassis with that in mind. Lastly, if you have DUAL CPU slots you might have issues. I don’t have that but from what I can tell, I think my card will block way for that as well.
  3. So 1070 has a ePCI 8 pin hole power cable. I tried to put the spare G0 / G1 cable which are normally plugged into the plastic frame  dicking around there, into 1070, but it won’t work and give you this: Which is kind of funny because if you found this post, you must have read other HP forum post bragging about the ePCI 5 pin cables are magical and has special properties and can support up to 225W GPU etc. They were right. BUT, you need to convert the 5pin (six pin with one empty) to a regular 8 pins for the GPU, and it will boot fine. So the answer is: if you want to use a GPU, get at least a 6 pin to 8pins converter or else it will not work for Z620. Directly plugging into the 6 pins into 8 pins holes on the graphics card will NOT WORK. The adapter cable does not have to come from HP or anywhere special, I bought it from a regular store.

Enjoy your new z620 with beefed up GPU card.

PS: for those who think this rig can handle SAS drives, think again and think very carefully. I bought tons of SAS drives and in the end have to return them because apparently my configuration at manufacturing at least did not support SAS drive unless I originally opted for an upgrade at the purchase time.

 
Posted in Uncategorized