/bin/sh: 1: ./XXXX.sh: not found

This obscure error plagued me a long time. This occurred in the context when trying to build a linux docker on a Windows host while trying to ask the Dockerfile to copy over a XXXX.sh file (name doesn’t matter) when it was originally created on windows and trying to execute it as RUN/CMD/ENTRY in the linux docker.

The problem is because of damn CRLF issue. Convert the XXXX.sh file into LF format and that will solve the problem.

See the source of the solution at https://github.com/postlight/headless-wp-starter/issues/171

 

Plotly Orca Bug and Workaround

TLDR: solution thread at https://stackoverflow.com/questions/58473837/plotly-missing-orca

I love plotly. I never remember any of their syntax when generating images and have to search every single time but they produce easily some of the most gorgeous graphs if you spend a few minutes, maybe hours, or days obsessing over the details and produce stunning images. I absolutely adore it for visualization.

However, Plotly has an unreliable dependency/connection with Orca. It is a PITA in the most utter, frustrating thing to work with and I struggled to solve it literally over span of months across dozens of computers and never managed to get it working… until now.

So, ORCA https://github.com/plotly/orca is the component that MUST be installed and does some black voodoo Javascript/NodeJS magic just so that… you can programmatically save the image generated in Plotly (at least, I believe that is its primary function). Yup. or you can just click on the button to save the image. However, this gets tedious when you have many images to save…

So in theory, it is easy as pie. Install, add binary to path, if not, run

plotly.io.orca.config.executable = '/path/to/orca' 

and everything should work. Except, it never worked across the half dozens of computer I tried on windows. In my jupyter notebook, I can even launch !orca but for some bizarre reason, the plotly fails to find ORCA executable WHILE printing out the path in the debug prompt. The fact that I POINT it to it via command above, still fails. Impressive…

So I thought I was special and must be something with my machine/env/windows/. Then I search and saw quite a few thread that leads to no where…

Eventually, I found the solution thread for ME at least: https://stackoverflow.com/questions/58473837/plotly-missing-orca. I need a shrine built for the user Susanna Ventafridda for finding this out.

In short, a MANUAL tweak to the Plotly package and alter the function call to WHICH will help find the package.

Not sure if this is a windows only issue or not. However, after MANUALLY updating that path to point to the orca.exe, it works for me at least… Don’t forget the r infront of the path string.

PS: Some additional annoyance I noticed right away:

  • Saving first image takes a few seconds to spin up the orca node server I presume on modern work stations…. oh the wonder of Javascript performance… for the task for saving an image… sigh.
  • fig.write_image() does not take Path… yeah… you have to pass it string path information. I feel dirty.
 

The Annoying PDF form that prevent editing after you fill it…

I ran into this bandit a few days back and was wondering what was wrong with my machine or the PDF I received it from.

Symptoms:

  • Open PDF, everything is editable.
  • After filling and saving it, the PDF form fields become NO LONGER EDITABLE, even in Adobe Acrobat.
  • Telltale sign: field of the Acrobat becomes GRAY instead of LIGHT BLUE.
    See below. The company name was previously fillable but no longer fillable. Regular fillable fields appear blue below. that.
  • Even going into Prepare Form Option to manipulate the field, they are not directly editable there short of deleting and redefine a form field.
  • Edit permission of the PDF is normal. You have full access to the PDF.

Fix:

  • Something triggered automatically 1) lock the fields of the form 2) set them to read only. I suspect it might be either LifeCycle at form design stage or a signature insertion during the form fill stage. Something caused it and I am not sure which one yet.
  • To fix it, unchceck the boxes in the right order in form preparation view (form editing view)

Screw whoever thought it was nice to make form “once fillable”. I had to Google some crazy crap people do to get around whoever blessed us with this cursed setting in the initial form design.

I should vote this to the /r/badUIbattles/

 

Interesting Observation on interpolate2d from numpy vs RBF from scipy.interpolate

So at work I have to convert a function from matlab interpolate2d to python. But there is no direct translation or rather, some similar but not identical implementation so I had to look it up. After running some unit tests and seeing some very bizarre value later, I stumbled across this insightful experimentataion post https://stackoverflow.com/questions/37872171/how-can-i-perform-two-dimensional-interpolation-using-scipy

After running some tests, I am seeing somewhere between 0.1 to 1% deviations between matlab vs python implementation. No big deal I thought. then, I realized that the input to matlab vs python was different, hence why the big deviation.

I figure I would apply the input correction to see again, BUT, i mad the mistake of incorrectly injected a wrong value in one of the input to python to construct the 2d interpolant. THIS is where things gets interesting, despitely WILDLY wrong value injection, the RBF moved from 0.1% error to 0.4% error. While interpret2D jumped from originally 1% deviation to about -800% ish…

Single outliers out of the 162+ pairs of points used to model the manifold seems to have a far far greater impact to interpolate2d than RBF.

 

Oi. My Ubuntu 18 suddenly give permission error mounting USB/M2/SATA Enclosure

This… is one of the most weird error I have seen in a while.

So I got this SATA M.2 enclosure hoping to reap the sweet sweet fast copy of a M.2 stick since I upgraded my laptop M.2.

The weird thing is… on this ONE particular Ubuntu 18.04 computer, for the love of !*()#_@!_ I could not mount the USB drive no matter what I try. It (NTFS) mounts fine on my other Ubuntu 18.04 PC, fine on PC laptop etc.

On this problematic computer, i get the error like: “Unable to mount volume Not Authorized to perform operation” when I already have sudo privilege. I initially thought it would be a driver issue but this cannot be as the other ubuntu OLDER actually mounts fine…

I reinstalled NTFS 3G and it made no differences.

Many many search of the error lead to this: https://askubuntu.com/a/1207432

In Ubuntu 18, this error is caused by Chrome Remote Desktop. FREAKING Chrome Remote Desktop manages to block my USB mounting? W.T.F…

The solution is:

  1. Stop the Chrome Remote Desktop Service/opt/google/chrome-remote-desktop/chrome-remote-desktop --stop

Mount the drive.

Then restart the service etc.

Expect to do this every time you restart… W.T.H.

Luckily I am not going to be using this particular USB device many times… Weird as heck bug tbh.

PS: this apparently is COMMON.

Related posts from online:

https://askubuntu.com/questions/1191603/unable-to-access-external-drives

https://askubuntu.com/questions/1196248/automount-when-clicking-on-disks-in-gui-file-manager-suddenly-stopped-working-in

 

Weird Plotly Express Bug…

Plotly 4.5.0 has always worked well for me but recently, I ran into this weird bug where it just crash out like a BAMF.

I wasn't even trying anything fancy, just an example code, on a BRAND NEW notebook: 

import plotly.express as px
data_canada = px.data.gapminder().query("country == 'Canada'")
fig = px.bar(data_canada, x='year', y='pop')
fig.show()

Then it puked its gut out more or less...


---------------------------------------------------------------------------
SyntaxError                               Traceback (most recent call last)
C:\ProgramData\Anaconda3\envs\bengali.ai\lib\site-packages\plotly\express\_core.py in apply_default_cascade(args)
    748         # retrieve the actual template if we were given a name
--> 749         args["template"] = pio.templates[args["template"]]
    750     except Exception:

C:\ProgramData\Anaconda3\envs\bengali.ai\lib\site-packages\plotly\io\_templates.py in __getitem__(self, item)
     90                     template_dict = json.loads(template_str)
---> 91                     template = Template(template_dict)
     92 

C:\ProgramData\Anaconda3\envs\bengali.ai\lib\site-packages\plotly\graph_objs\layout\__init__.py in __init__(self, arg, data, layout, **kwargs)
   9449         _v = arg.pop("data", None)
-> 9450         self["data"] = data if data is not None else _v
   9451         _v = arg.pop("layout", None)

C:\ProgramData\Anaconda3\envs\bengali.ai\lib\site-packages\plotly\basedatatypes.py in __setitem__(self, prop, value)
   3481             if isinstance(validator, CompoundValidator):
-> 3482                 self._set_compound_prop(prop, value)
   3483 

C:\ProgramData\Anaconda3\envs\bengali.ai\lib\site-packages\plotly\basedatatypes.py in _set_compound_prop(self, prop, val)
   3835         validator = self._validators.get(prop)
-> 3836         val = validator.validate_coerce(val, skip_invalid=self._skip_invalid)
   3837 

C:\ProgramData\Anaconda3\envs\bengali.ai\lib\site-packages\_plotly_utils\basevalidators.py in validate_coerce(self, v, skip_invalid)
   2443         elif isinstance(v, dict):
-> 2444             v = self.data_class(v, skip_invalid=skip_invalid)
   2445 

C:\ProgramData\Anaconda3\envs\bengali.ai\lib\site-packages\plotly\graph_objs\layout\template\__init__.py in __init__(self, arg, area, barpolar, bar, box, candlestick, carpet, choroplethmapbox, choropleth, cone, contourcarpet, contour, densitymapbox, funnelarea, funnel, heatmapgl, heatmap, histogram2dcontour, histogram2d, histogram, image, indicator, isosurface, mesh3d, ohlc, parcats, parcoords, pie, pointcloud, sankey, scatter3d, scattercarpet, scattergeo, scattergl, scattermapbox, scatterpolargl, scatterpolar, scatter, scatterternary, splom, streamtube, sunburst, surface, table, treemap, violin, volume, waterfall, **kwargs)
   1582         _v = arg.pop("parcoords", None)
-> 1583         self["parcoords"] = parcoords if parcoords is not None else _v
   1584         _v = arg.pop("pie", None)

C:\ProgramData\Anaconda3\envs\bengali.ai\lib\site-packages\plotly\basedatatypes.py in __setitem__(self, prop, value)
   3485             elif isinstance(validator, (CompoundArrayValidator, BaseDataValidator)):
-> 3486                 self._set_array_prop(prop, value)
   3487 

C:\ProgramData\Anaconda3\envs\bengali.ai\lib\site-packages\plotly\basedatatypes.py in _set_array_prop(self, prop, val)
   3909         validator = self._validators.get(prop)
-> 3910         val = validator.validate_coerce(val, skip_invalid=self._skip_invalid)
   3911 

C:\ProgramData\Anaconda3\envs\bengali.ai\lib\site-packages\_plotly_utils\basevalidators.py in validate_coerce(self, v, skip_invalid)
   2531                 elif isinstance(v_el, dict):
-> 2532                     res.append(self.data_class(v_el, skip_invalid=skip_invalid))
   2533                 else:

C:\ProgramData\Anaconda3\envs\bengali.ai\lib\site-packages\plotly\graph_objs\__init__.py in __init__(self, arg, customdata, customdatasrc, dimensions, dimensiondefaults, domain, ids, idssrc, labelangle, labelfont, labelside, line, meta, metasrc, name, rangefont, stream, tickfont, uid, uirevision, visible, **kwargs)
  43792         _v = arg.pop("line", None)
> 43793         self["line"] = line if line is not None else _v
  43794         _v = arg.pop("meta", None)

C:\ProgramData\Anaconda3\envs\bengali.ai\lib\site-packages\plotly\basedatatypes.py in __setitem__(self, prop, value)
   3481             if isinstance(validator, CompoundValidator):
-> 3482                 self._set_compound_prop(prop, value)
   3483 

C:\ProgramData\Anaconda3\envs\bengali.ai\lib\site-packages\plotly\basedatatypes.py in _set_compound_prop(self, prop, val)
   3835         validator = self._validators.get(prop)
-> 3836         val = validator.validate_coerce(val, skip_invalid=self._skip_invalid)
   3837 

C:\ProgramData\Anaconda3\envs\bengali.ai\lib\site-packages\_plotly_utils\basevalidators.py in validate_coerce(self, v, skip_invalid)
   2443         elif isinstance(v, dict):
-> 2444             v = self.data_class(v, skip_invalid=skip_invalid)
   2445 

C:\ProgramData\Anaconda3\envs\bengali.ai\lib\site-packages\plotly\graph_objs\parcoords\__init__.py in __init__(self, arg, autocolorscale, cauto, cmax, cmid, cmin, color, coloraxis, colorbar, colorscale, colorsrc, reversescale, showscale, **kwargs)
   1417         _v = arg.pop("colorbar", None)
-> 1418         self["colorbar"] = colorbar if colorbar is not None else _v
   1419         _v = arg.pop("colorscale", None)

C:\ProgramData\Anaconda3\envs\bengali.ai\lib\site-packages\plotly\basedatatypes.py in __setitem__(self, prop, value)
   3481             if isinstance(validator, CompoundValidator):
-> 3482                 self._set_compound_prop(prop, value)
   3483 

C:\ProgramData\Anaconda3\envs\bengali.ai\lib\site-packages\plotly\basedatatypes.py in _set_compound_prop(self, prop, val)
   3835         validator = self._validators.get(prop)
-> 3836         val = validator.validate_coerce(val, skip_invalid=self._skip_invalid)
   3837 

C:\ProgramData\Anaconda3\envs\bengali.ai\lib\site-packages\_plotly_utils\basevalidators.py in validate_coerce(self, v, skip_invalid)
   2443         elif isinstance(v, dict):
-> 2444             v = self.data_class(v, skip_invalid=skip_invalid)
   2445 

C:\ProgramData\Anaconda3\envs\bengali.ai\lib\site-packages\plotly\graph_objs\parcoords\line\__init__.py in __init__(self, arg, bgcolor, bordercolor, borderwidth, dtick, exponentformat, len, lenmode, nticks, outlinecolor, outlinewidth, separatethousands, showexponent, showticklabels, showtickprefix, showticksuffix, thickness, thicknessmode, tick0, tickangle, tickcolor, tickfont, tickformat, tickformatstops, tickformatstopdefaults, ticklen, tickmode, tickprefix, ticks, ticksuffix, ticktext, ticktextsrc, tickvals, tickvalssrc, tickwidth, title, titlefont, titleside, x, xanchor, xpad, y, yanchor, ypad, **kwargs)
   1800         _v = arg.pop("tickfont", None)
-> 1801         self["tickfont"] = tickfont if tickfont is not None else _v
   1802         _v = arg.pop("tickformat", None)

C:\ProgramData\Anaconda3\envs\bengali.ai\lib\site-packages\plotly\basedatatypes.py in __setitem__(self, prop, value)
   3481             if isinstance(validator, CompoundValidator):
-> 3482                 self._set_compound_prop(prop, value)
   3483 

C:\ProgramData\Anaconda3\envs\bengali.ai\lib\site-packages\plotly\basedatatypes.py in _set_compound_prop(self, prop, val)
   3835         validator = self._validators.get(prop)
-> 3836         val = validator.validate_coerce(val, skip_invalid=self._skip_invalid)
   3837 

C:\ProgramData\Anaconda3\envs\bengali.ai\lib\site-packages\_plotly_utils\basevalidators.py in validate_coerce(self, v, skip_invalid)
   2440         if v is None:
-> 2441             v = self.data_class()
   2442 

C:\ProgramData\Anaconda3\envs\bengali.ai\lib\site-packages\plotly\graph_objs\parcoords\line\colorbar\__init__.py in __init__(self, arg, color, family, size, **kwargs)
    694         # -----------------
--> 695         from plotly.validators.parcoords.line.colorbar import tickfont as v_tickfont
    696 

SyntaxError: invalid character in identifier (__init__.py, line 1)

During handling of the above exception, another exception occurred:

ValueError                                Traceback (most recent call last)
<ipython-input-2-11d79a4bcb46> in <module>
      1 import plotly.express as px
      2 data_canada = px.data.gapminder().query("country == 'Canada'")
----> 3 fig = px.bar(data_canada, x='year', y='pop')
      4 fig.show()

C:\ProgramData\Anaconda3\envs\bengali.ai\lib\site-packages\plotly\express\_chart_types.py in bar(data_frame, x, y, color, facet_row, facet_col, facet_col_wrap, hover_name, hover_data, custom_data, text, error_x, error_x_minus, error_y, error_y_minus, animation_frame, animation_group, category_orders, labels, color_discrete_sequence, color_discrete_map, color_continuous_scale, range_color, color_continuous_midpoint, opacity, orientation, barmode, log_x, log_y, range_x, range_y, title, template, width, height)
    311         constructor=go.Bar,
    312         trace_patch=dict(orientation=orientation, textposition="auto"),
--> 313         layout_patch=dict(barmode=barmode),
    314     )
    315 

C:\ProgramData\Anaconda3\envs\bengali.ai\lib\site-packages\plotly\express\_core.py in make_figure(args, constructor, trace_patch, layout_patch)
   1321 
   1322 def make_figure(args, constructor, trace_patch={}, layout_patch={}):
-> 1323     apply_default_cascade(args)
   1324 
   1325     args, trace_specs, grouped_mappings, sizeref, show_colorbar = infer_config(

C:\ProgramData\Anaconda3\envs\bengali.ai\lib\site-packages\plotly\express\_core.py in apply_default_cascade(args)
    750     except Exception:
    751         # otherwise try to build a real template
--> 752         args["template"] = go.layout.Template(args["template"])
    753 
    754     # if colors not set explicitly or in px.defaults, defer to a template

C:\ProgramData\Anaconda3\envs\bengali.ai\lib\site-packages\plotly\graph_objs\layout\__init__.py in __init__(self, arg, data, layout, **kwargs)
   9429 The first argument to the plotly.graph_objs.layout.Template
   9430 constructor must be a dict or
-> 9431 an instance of plotly.graph_objs.layout.Template"""
   9432             )
   9433 

ValueError: The first argument to the plotly.graph_objs.layout.Template 
constructor must be a dict or 
an instance of plotly.graph_objs.layout.Template

I fixed the issue by pip uninstall and reinstall plotly 4.5.0.

Who says turn off and turn back on doesn’t work in software world ^o^.

 

The annoying bug of scp silently fail without error message.

I was totally thinking I did something terrible wrong when trying to copy files through a jump server. It turned out, to be nothing related to jump server but apparently a known issue with SCP silently failure when something funny going on with .bashrc It took a few searches to lead to these results and hopefully it will help you too.

The solution is suggested here: https://www.fir3net.com/UNIX/General/why-does-scp-file-transfer-fail-but-there-is-no-error.html

# If not running interactively, don't do anything
[[ $- == *i* ]] || return

Also detailed here in StackOverflow: https://unix.stackexchange.com/questions/18231/scp-fails-without-error

 

It is probably not the best idea to use CPU_FAN_OPT for anything other than CPU fan…

Take this with massive NaCl crystals… as I am definitely not an expert in this area, but… I believe I *might have resolved some odd issues my AMD Wraith prism (which came with the Ryzen 7 2700X)was running into.

I put this computer together about last Prime day, and worked reasonably well… except sometimes, suddenly, all/many/some fans stops working. As in, completely, shut off.

Other times, I left the computer on for sometimes and come back to find the computer turned off by itself. After using the computer for a while, I personally observed several times where the CPU fan STOPPED working while I am in the middle of things, not necessarily computationally intensive stuff either. It was super weird. I was suspecting a faulty Wraith Prism… but come on, a brand new fens, how complicated it can be. Then, it happened several time where thermal shutdown actually happens due to excessive heat from the CPU (i.e. thermal protection kicked in to force shutdown)…. No doubt that shortened my CPU life by quite a few cycles..

Another thing I observe was that… my CPU was running hotter than probably most other 2700X reported. I am at around 55+ to sometimes 65+. Some say it is normal but I figure, it is probably a overclocked chip so it should run hotter than that… Big deal…

Then, today, I finally had enough and decided to see what is up. My motherboard is ROG Crosshair VI Extreme (a pretty old mobo). The CPU fan (wraith prism) is plugged into the CPU_FAN slot as intended… so nothing super unusual. Then, I saw I plugged a case fan extension card into the CPU_FAN_OPT card slot. Normally, I figure that an additional extension card on a regular fan out slot cannot be that bad… so I decide to move the card off from the CPU_FAN_OUT from there to a radiator/fan header instead.

Then when I put the computer back together, instantly, I am seeing my 2700X chilling at around 38C instead of the regular 50/60+. BAM. ten degree deduction at least. Also noticeably, the fan is spinning faster as well (normally, I can see the individual fan leaf)… So while correlation doesn’t not imply causation, I am happy with the current configuration and will DEFINITELY avoid plugging anything into the CPU_FAN_OPT unless it is actually another CPU related fan…

My suspicion is that maybe the CPU_FAN_OPT and CPU_FAN are working in conjunction and when one is drawing too much power, the other might have reduced current/voltage. If my other case fan are spinning hard, it might reach low enough current to the main wraith prism to cause it to stop. I can probably run another experiment to see but I am quite lazy and so far very happy with the way this is turning out… and hopefully better service time of my computer… I will update should I run into the shut off issue further down the road.

 

Docker tips, tricks and gotchas…

I have been doing some test and exploration putting Docker containers together via docker-compose. Here are some tricks not mentioned in most cheat sheets but are very valuable in my journey as I struggle with it. I will keep trying to update this.

Networking: 

  • Container IP:
    • docker inspect -f "{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}" 
    • then add 4 alphanumeric of the containers ID to get the IP.
  • Apache in Docker container needs to be run using foreground process for it to work :e.g.
    • CMD apachectl -D FOREGROUND
  • A common problem I run into a LOT is, inter docker communication. Which seems SHOULD be straight forward but insanely annoying to debug somehow.
    • nslookup from the dnsutils repo seems to help clear the fog somewhat to ensure at least they can NameService resolve to other containers!

Management:

  • Always use docker ps -a instead of docker ps  as the later does not show stopped containers. 
  • docker rm is for container and docker rmi is for images
    • If you are developing and have to debug image/container as part of the compose, best to have a simple bash script like this to quickly remove image/container:
      • #!/usr/bin/env bash
        # bach script to automatically remove the image/container to
        docker rm DC_LORIS22Configurator_v0.4
        docker rmi dockerbuilds_chrome:latest
    • Alternatively, force newer image and ignore cache. HOWEVER, this is non-selective i believe so if you have a compose with many images, it might be more efficient to go first.
          --no-cache              Do not use cache when building the image.
          --pull                  Always attempt to pull a newer version of the image.
  • To build a custom Docker image file, make sure to specify the context
    • docker build -f Dockerfile.LORIS21.Ubuntu18 ../../
    • If the context is not specified properly, the copying file operations might fail.
  • To run bash when the docker file declared an entry point, try things like below to bypass the entry point:
    • docker run -it –entrypoint /bin/bash <image> 

Docker File:

  • RUN vs CMD and WAIT for​_ service to be up before running:
    • When docker compose run all things, they don’t care about whether it is fully ready ready. They launch and assume ready… The Container needs to be robust enough to check this before fully run. One dumb/lazy way to solve this is to delay the launch. E.g. at ENTRYPOINT/CMD ensure there is a sleep 1 minute or something. DO NOT DO THIS IN RUN as that is executed at image building and will not affect deployment whatsoever…
  • This is the defacto guide on Docker ARG/ENV/.ENV variables https://vsupalov.com/docker-arg-env-variable-guide/
    • My approach was actually set ARG using .env at DockerCompose level, use ARG to set ENV at DockerFile level, and then in my app/script load ENV. This seems SUPER convoluted but it give the flexibility that you can reuse DockerFile (via ARG), reuse app/script via ENV variable, and have a lot flexibility but maintaining variable naming consistency can be annoying .
  • One issue I found commonly cause my ARG (loaded from Compose/ENV) to not work is because I forgot to declare the ARG in its own line ahead of time….

Docker Compose:

  • This stackoverflow thread give a very good quick insight into how to persist volumes in the Docker Compose context and Docker File context: https://stackoverflow.com/questions/40905761/how-do-i-mount-a-host-directory-as-a-volume-in-docker-compose 
  • Make sure to wait for services to fully start (i.e. depend_on merely LAUNCHES services, not when ready.) before calling them.
  • When specifying volume for services, make sure there is no space after column
    volumes:
    - lorisdbvolume:/var/lib/mysql

Debugging:

  • Ctrl + P + Q puts interactive Docker sessions to background
  • Docker stop all container:
    • docker stop $(docker ps -a -q)
  • Avoid debugging dockerfile and dockercompose at the same time.
    • Get dockerfile working by itself before putting them together.

MySQL

  • Not exactly docker related but I stuck on this while trying to connect to mysql in a docker bash for a long long time: the proper syntax is something like this:
    • mysql $LorisMySQLDatabase -hlorismridb -p3306 -u$LorisMySQLUser -p$LorisMySQLUserPassword
    • Notice that port cannot be part of the host name, at it will not resolve properly and lead to MySQL Client Error 2005/2003.
    • Use bash environmental variable (after securing the container).
    • Note that the h/p/u/p flags are always immediately proceeding.