Azure CLI – az login – behind SSL intercepting DLP/proxy

https

Should you be in similar situation to many of us behind corporate firewall intercepting SSL connections (i.e. DLP/Layer 7 inspection) trying to use Azure CLI (az) will raise following error:

# az login
Please ensure you have network connection. Error detail: HTTPSConnectionPool(host='login.microsoftonline.com', port=443): Max retries exceeded with url: /common/oauth2/devicecode?api-version=1.0 (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')],)",),))

Above will be shown even if system wide SSL CA Certificate used by intercepting proxy is installed on your system.

Normally to get such certificate installed for system wide use, following needs to be done:

  1. copy certificate to: /usr/local/share/ca-certificates/<your-ca>.crt
  2. run, i.e. on debian/ubuntu flavours: update-ca-certificates
  3. relogin

This though won’t work with Azure CLI as it uses Python related certificate chains ignoring the system wide.

The easiest workaround is to force Azure CLI to use the system wide SSL trusted certificate file:

# echo "export REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt" >> ~/.bashrc

Once above is done – re-login and run az login – all should work fine.

Hope this helps.

OpenVPN & Office 365 / Windows 10 nightmare

"We are unable to connect right now. Please check your network and try again later."
-- your beloved vendor

Yep, we’ve all seen it and if you’re here reading it, it only means you’ve got enough of it and it is time to resolve Microsoft Windows Known Issue

OpenVPN and other VPN solutions allow to push all traffic through VPN gateway. All good so far, right? To do that default gateway is set.

Again, if you’re here it means that the problem you’ve run into with connectivity “issue” is not a real connectivity issue as you’ve certainly established that even if this message is shown, you’re able to browse internet using browser, icmp/traceroute shows traversing correct path via vpn gateway, etc.

So what the heck is going on here and why Office 365 is so stubborn with its claim?

The problem originates from the fact that Office 365 relies on Network Location Awarness (NLA) which uses Network Connection Status Indicator (NCSI) to decide if “connection” to internet works for it’s call home functions.

The inner works of NCSI is that it goes out and tries to get to following in order to verify “Internet connectivity” http://www.msftncsi.com/ncsi.txt. Some reports show also following as probed dns.msftncsi.com, 131.107.255.255.

None of that sounds scary, all reasonable right?

To make sure that traffic is going through VPN gateway, you’ve checked already traceroute (tracert).

The setting to have all traffic pushed via gateway on OpenVPN is:

push "redirect-gateway def1"

That’s where a lot of us were stuck, myself for a year, returning to it from time to time. At one point in time, I thought that M$ needs to check “licensing” and it is some sort of smartness/dumbness there that it does this as first thing whilst starting Windows, prior you get any control and sees its proper public address. To make it worse in troubleshooting I’ve noticed a lot of SYN followed by RST/RST,ACK when connected via VPN and when going out through clear connection it was nicely setting up tcp session. This in itself made me thinking that it is some sort of multi-layer communication, where first connection pull something and prompts Microsoft servers to accept following connections from given IP and as result, after connecting to VPN, I was getting RST and not able to connect.

This was until today – got upset with that problem and spent additional time on testing, redirecting traffic, resetting status, etc. This brought me to discovery of following.

Default gateway or not…?

Looking at ipconfig output and Network and Sharing Center, I’ve discovered that:

a) Network Sharing shows VPN/OpenVPN connection as “No Internet”

b) there is no default gateway on the output of ipconfig.

Hold on… no default gateway? How the heck my whole internet access via VPN (ok outside of Office 365 works then)? No, no proxy, no hijack, no nothing. Checking routing table – all looks ok’ish – as always, VPN routes, + 127.0.0.0 route, etc. Not so good metrics but everything else works.

What the heck? – Solution

This led me to small test – how about if I’d add default gateway manually or would not rely on OpenVPN “redirect-gateway def1” setting? Going ahead with:

push "route 0.0.0.0 0.0.0.0"

in ccd (client configuration definitions?) of OpenVPN sorted out the thing. As simple as that.

Side info, ccd file is named after host certificate as it connected to OpenVPN. Settings equally could be pushed out to all connecting clients, but that wasn’t my requirement for other reasons.

This helped to get “Internet” access under Network and Sharing Center and see default gateway on ipconfig output.

Victory!

References:

  • https://answers.microsoft.com/en-us/windows/forum/windows_10-networking/network-connection-status-indicator-ncsi-showing/02664ddf-4eac-449a-8318-bdae1a5bad3d
  • https://www.macwheeler.com/windows-10-office-365-cannot-connect-over-openvpn-fixed/ – found it after finding solution, though it shows enforcing it from client side using Windows UI.

VMFS-6 read file error – Found stale lock

Once upon a time there was power cut and ESXi system went down.

… for these in rush, scroll down to “Fix” section…

After power-up none of details could be seen for VM which was up at the time of power cut. On VMs list page it was showing VMFS path to the file and that was it.

CLI level investigation shown that neither VM.vmx nor VM.vmdk files could be read and were raising error.

Fun…. fun… fun….

Disclaimer:
All below is against any good practice and/or anything Vmware would recommend as overwrites sensitive meta-data information of vmfs.
With that said, Vmware does not provide any way to fix it as of today and the only way would be to contact Vmware support.
All below is for educational purposes only and done at your own risk.

First steps

First steps were towards voma tool and output was similar to below (though below is taken from internet as no screenshots were taken at the time of the issue):

voma -m vmfs -f check -d <path_to_device>

returned output similar to below:

#https://www.reddit.com/r/vmware/comments/9j8k6o/esxi_vmfs6_datastore_corruption_after_host_reboot/

VOMA unfortunately does not support VMFS-6 in fix mode (as of 2018.11.02 on ESXi 6.5 and 6.7).

vmfs-tools (https://glandium.org/projects/vmfs-tools/) does not support VMFS-6 neither.

This left me in cul-de-sac… almost.

Big, big thanks to: Ulli aka continuum at communities.vmware.com helped to solve the issue.

Fix

1.
Dump heartbeat section of your VMFS-6 in question (.vf.sf file is in root folder of VMFS-6)

2.
Verify if the file contain only locks from your system (needs to be done on other system as strings is not available on ESXi, scp or any other way to get the file out of your system is your friend) :

3.
If above is confirmed, generate a clean heartbeat section using same build of ESXi and dump it to file:

4.
Transfer that clean file to your ESXi server and incorporate it into VMFS-6 with issues:

Within a minute or two earlier locked files should be accessible if not, try to reboot your ESXi.

 

Links:

Locked files with VMFS 6

Create a VMFS-Header-dump using an ESXi-Host in production

https://communities.vmware.com/thread/597513

Apple censorship – posts on LSIs (liquid indicators) removed by Apple

Dear,

There’s a lot of noise about Apple products and how Apple refuses to accept warranty repairs abusing LSIs.

Apple went one step further and censors and tries to mute all discussions about this subject.

Below is example of post they removed claiming is “inappropriate” for forum whilst was posted under thread where LSI was subject.

Original post below.

Hi (SecTec),

Thanks for participating in the Apple Support Communities.

We’ve removed your post macbook pro problem – “water damage”- Really? because it contained either product feedback or a feature request that was not constructive.

To read our terms and conditions for using the Communities site, see this page:  Apple Support Communities  – Terms of Use

We hope you’ll keep using our Support Communities. You can find more information about participating here:  Apple Support Communities  – Tutorials

If you have comments about any of our products, we welcome your feedback:  Apple – Feedback

We’ve included a copy of your original post below.

Thanks,
Apple Support Communities Staff


Original – removed post:

I’ll add to it a bit of my story, same as above which just highlights that there’s something wrong with MacBook Pro design and/or LSIs (liquid indicators).

 

The story started back in 2014, bought new MacBook Pro, best available model. Used it for professional use, traveling a lot by plane, fully aware of how to operate computer equipment as I used to assemble PCs in the past and am in industry for 20+ years. Worth noting that prior MacBook Pro, used all sorts of other vendors, Dell, Lenovo, HP – never had any problems with equipment and kept replacing it only when it was too old/slow.

 

Back to main story, the acquired MacBook was put into Speck enclosure to protect it from body damages, this was ca. September 2014.

I have also additional insurance covering any liquid spills, etc. so essentially I’m not bothered with that from expenses side, but more from the aspect that I’ve never spilled any liquid on any of my laptops. I’ve learnt my lesson with coffee over keyboard back in the past with desktop PCs, so very careful these days and never have any food/liquids around laptops.

 

Fast forward to January 2015, was traveling to Florida, this time for leisure. Rarely used system, however on the very last day was checking flight plan for the day when the SSD died. System did hang and upon a try to reboot reported missing drive.

Was delivered to Service center. Forced technician to start tests in front of me which didn’t make them happy. There were no complaints about any damages on the body. Had to leave system with them.

Up to my huge surprise couple of days later received a call that LSIs were triggered.

I’ve requested to see that and at that time discovered that:

a) usb port was damaged (sic!), technician was claiming that it was since the beginning,

b) seen LSI triggered (red).

 

What came to my attention was that only one LSI was triggered and none other. This did lead me to request technician how in the world this one could be triggered if none other was triggered.

This did lead him to seek Apple authorization for exceptional approval.

The SSD has been replaced in result.

 

In April 2014, system became slow (like really slow, think about PC with i396 CPU) and started to report that battery needs replacement, was powering off seconds after disconnecting charger.

 

System was then delivered to Apple store. System was inspected by technician and I’ve received call that system was exposed to water and that stops any further activities from their side.

It took some effort to collect documentation from previous case to present them that nothing did happen since last repair, no other LSIs were triggered and this technician also admitted that it is not the first time he has seen LSIs triggered which would be difficult to have triggered as single one and no others.

System was repaired, this time battery, meaning the whole lower body due to new battery specifics.

 

Moving forward to December 2015, the very same Mac failed again. Same issue as previously, meaning slow system, charging.

 

This meant for me couple of things,

a) previous slowness happen just before business trip, but as it was night before, I thought it was just a slow system and battery issue, I took Mac with me, only to discover that it was useless. This forced me to buy new system with similar specification as Mac, due to my profession I have to have systems with 16GB ram, SSD, etc. So, fun, fun… a lot of expenses.

b) this time it failed during business trip and guess what, I didn’t have the earlier acquired spare system with me. Due to importance of the trip, length and requirements, I’ve had to acquire yet another system (sic!). In total two spare systems, thank you Apple.

c) system was out of standard Apple warranty and due to above, I was sick with issues with this system, I’ve submitted claim to have system replaced .

 

Couple of days later, I’ve received call from technician asking questions about the issue, history, etc. Good, they started to work on it.

Up to my surprise there was no sign of life from their side for very long time, when they called back it was 20 days after reporting issue to them.

Without any surprise I’ve been told that there were LSIs in red and as result they deny any of my claims due to water, etc. This was certainly a nice excuse and try from their side, as these repairs are on Seller cost and not necessarily considered as “warranty” repair. I’m unsure how Apple plays with their cost centers, etc… it is outside of this story.

 

Their push back was so mean that it triggered me to not exercise my insurance

 

It took a nice additional days to get it executed.

In March 2016, I’ve had new MacBook Pro in hand.

 

I was so ****** by that time and so used to my new Lenovo, that I kept it using, happily… no issues.

MacBook was happily sleeping in unsealed, original Apple box.

 

For some reason I’ve decided to start using the system again around December, as the retina display provide nice colors comparing to Lenovo display.

 

So, was happy Mac user again.

 

Just until March 2017…. so it counts at max 3 months!!!

 

Mac failed again. This time it had close to zero trips, stayed at home. This time it failed in different way, was left at the desk, went to sleep (display off). The next day I’ve tried to wake it up, no reaction. Tried all combinations SMC reset, etc. Noticed that Mag plug was showing green indicator. Since MacBook are so great in communication with user about it’s state it was my only indicator to see if it reacts on anything. Unplugged it, plugged back and zero, nothing, no lit… just gray… no orange, no green.

Ok… now it is dead like never again. Called Apple, they pushed me through standard, check connector, etc…

 

Went to Service Center again, technician happily noted, no scratches, nothing, mint condition. Given past experience I’ve pushed them to sign that no physical damages were there. Since I was in rush, it was late and I’ve had to prepare for trip next day, couldn’t wait or go to Service Center when they would be able to open system next to me.

 

The day after, I’ve received mail from Service Center. 800$ for Motherboard, 1000$ for 512GB SSD + some peanuts for fans.

As per Apple Technician, one LSI is on and up to my bigger surprise they claim there were traces of some liquid inside!!! I can honestly say that this system didn’t have any contact with any liquid. I’ve been super duper careful with it, much more with any other system earlier, due to earlier experience.

Yet, Apple once again claims system had contact with liquid.

 

I’m sorry Apple, it is either a scam or something is wrong with design and/or LSIs. What I can say, there was no liquid spilled. The conditions in which system was used, was always in-house, office, hotel, living room, lounge at airport, plane. No, no bathroom, no kitchen, no exteriors, nothing. Always dry air, no temperature shock, system was always in sleeve and then in travel rolling bag (very well protected from everything).

Leaving system at Apple didn’t think they could say anything about LSI, especially this time when I took care of the system that much.

 

Interesting in the whole story is that at the same time, the Lenovo system I’m typing from now, was used for much more time than the last MacBook. Earlier I’ve used the good old ThinkPad W530, etc. never had an issue.

 

What did I like about MacBook pro?… that was hardware design as it nice, lightweight and small comparing to T530 tank.

As this MacBook failed again, am I going to stay with Apple? Rather not…

I’ll try to understand what LSIs were triggered and will see what Apple has to say, but for the sake of time, this time I’ll probably exercise the insurance unless Apple will be helpful.

Depending on that I’ll either walk away from Apple or might stay… not really sure.

 

This will have some, potentially minor impact on Apple sales, but some will be there, I’ll advise internally within company to stop purchasing Apple, so we talk about potentially another 200-300 units which would be otherwise purchased within next 3 years or so.

I can’t recommend to any business now to use Apple.

What could change that is Apple admitting some design issues and take it more seriously.


This is a send-only account. Replies received at this address are automatically deleted.


TM and copyright © 2017 Apple Inc. 1 Infinite Loop, MS 96-DM. Cupertino, CA 95014.

OpenVPN sudo and pam failure

Problem comes from systemd new setting on 17.04+ (experienced on 18.04):

Example:

The fix is to run:

# paste

Write changes. This creates file:

Now we need to reload:

 

Fix has been mentioned:

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=792653#25 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=792653#45

 

 

 

SATA passthrough on ESXi 6.5 & 6.7

lspci -v|grep -i Mass -A1
0000:00:17.0 SATA controller Mass storage controller: Intel Corporation Sunrise Point-LP AHCI Controller [vmhba0]
Class 0106: 8086:9d03

vi /etc/vmware/passthrough.map

8086 9d03 d3d0 false

reboot

Works on ESXi 6.5, didn’t work on ESXi 6.7 (fresh install in both cases).

Backup IMAP account

The need behind this scenario is very simple – make sure I have up to date backup of my IMAP account as sometimes it does happen that access to the account might be lost for whatever reason.

Additional requirement is to have it “just working” without any activity from my side.

What didn’t work very well was Thunderbird synchronizing all emails for offline use. The reason behind was that structure of the account had moving parts – new subfolders appearing and being moved to other locations. With Thunderbird it did require manual selection of “synchronized” folders which was a daunting task.

The solution is to use mbsync, synchronizing from remote IMAP server to local folder which is then exposed via local dovecot for use with any IMAP client. In this particular situation Thunderbird is used, though is set to not cache any emails older than 1 day (minimum setting within Thunderbird).

How to set it up

mbsync

In my particular situation, I’ve had to build isync on my own, as the repository contained old (v1.1) version vs currently available (v1.3).

To build isync following steps were used:

This built our needed isync_1.3.0-1_amd64.deb

File ~/.mbsyncrc was created with following

The aim was simple – dump and maintain copy if IMAP and follow any removals of emails, etc.

Folder itself is backed up incrementally with borg backup for history, just in case if someone would remove all content on remote side. This would propagate nicely removing all emails on local side and borg backup allows us to revert to previous stage.

Mbsync finishes its task once goes through all folders and does not maintain connection for updates. Therefore we need to run it periodically from cront

With this we will have periodically synchronized offline repository of emails (one way – to our local folder only).

Dovecot

Dovecot will serve our local folder via IMAP to any client. This is dictated by the fact that Thunderbird doesn’t play well with maildir folder and I couldn’t get it to discover new emails appearing in folders. I would have to have a dirty solution to remove “.msf” files as it forced TB to discover new files. Though problem did persist with discovering folder structure.

Dovecot installation and configuration is very simple:

Update /etc/dovecot/dovecot.conf :

Update /etc/dovecot/conf.d/10-mail.conf

 

Update /etc/dovecot/conf.d/20-imap.conf in section

New dovecot packages uses upstart, hence “start/stop/restart dovecot” commands need to be used to restart process instead of /etc/init.d/dovecot restart, etc.

Thunderbird

Standard Thunderbird setup with just one minor modification.

Deselect in Server Settings -> Advanced -> Show only subscribed folders

For whatever reason the subfolders are not under Inbox. It has something to do with Dovecot configuration, but since this worked for me I didn’t waste more time to investigation. I didn’t need to subscribe to all folders as I was accessing my local offline copy of files already.

Just to be on safe side and avoid email duplication under Account -> Synchronization & Storage -> Keep messages for this account on this computer” has been unchecked.

ssh to old HP iLO

Connecting to HP running some older version, only used internally, one can run into situation where after upgrade of local client it is not possible anymore to connect to managed HP ILO system.

Message presented is:

A bit or research it came out that ssh has disabled some not so secure read weak combinations which resulted in that problem.

To workaround it, just follow recommendation from http://www.openssh.com/legacy.html

Below is copy paste from this site.

or in the ~/.ssh/config file:

Soon after another problem was revealed:

To which there’s solution too.

OpenSSH 7.0 and greater similarly disable the ssh-dss (DSA) public key algorithm. It too is weak and we recommend against its use. It can be re-enabled using the HostKeyAlgorithms configuration option:

or in the ~/.ssh/config file:

It’s also possible to query the configuration that ssh is actually using when attempting to connect to a specific host, by using the -G option:

Hope this helps.

Mint on Dell Precision 5520 – fan noise

The key was the patch:

@Credits go to someone as when downloading didn’t take a note and don’t have time to look for it.

The important part is that both smm from this package is required as well as module after each kernel has to be rebuilt (make) and installed in appropriate location

Files I’m using to disable (obligatory to have i8k running as otherwise can burn laptop). Content of /usr/local/sbin/fan-bios-disable

Content of /usr/local/sbin/fan-bios-enable

And /etc/i8kmon.con

cpufreqd might be a good addition to control governors and maximum frequency, i.e. force lowest during the night to avoid any fan.

Disadvantage is that at least brightness control doesn’t work.

Searching for more details here can be found:

XPS 9560 – Battery life optimization and fan management from Dell

Great help was found at https://github.com/vitorafsr/i8kutils/issues/6

Ubuntu/Mint/Debian btrfs compressed at installer time

Easiest way to do this is to alter the mount command of the live environment.

Boot as usual to the live session.

Move the mount executable to another location:

Edit a new file using sudoedit /bin/mount and save the following script into it (alter the options as you like; here we have added compress):

 

You can also match block devices like /dev/sda1 instead of -t btrfs and chain elifs to use different mount options for different devices and filesystems.

Copy the original permissions over to the new script:

Install as usual and your btrfs partition will be mounted with the specified options (here, compress).
After the installation is finished, before exiting the live envirement, alter the /etc/fstab of the newly installed system to match the specified options, so it will use the same options on new boots.

Additional options one might consider to add are:

One more seen online was realtime

@Credits to someone out there – not referenced to exact source as it comes from my notes.