Blog

Nextcloud SSO with Keycloak

2022-08-27T16:00:00-05:00

This documents how I configured Nextcloud to use Single Sign-on leveraging Keycloak as the identity provider. There is official documentation for a Nextcloud-Keycloak integration, but it is behind Nextcloud's paywall.... I found several other guides, some directly conflicting with each other, but piecing them together has produced a working solution.

On to getting things working:

Nextcloud configuration

Global Settings

First, ensure that you do not lock yourself out of your Nextcloud instance while configuring SSO! Set the "Global Settings" to allow multiple back-ends.

Service Provider Data

Provide a certificate for request signing. This is not publicly seen; it is only used between Nextcloud and Keycloak. I opted for a self-signed x.509 certificate. It will be imported to Keycloak as part of the metadata XML so trust will not be an issue. Fill in the certificate request using whatever information you wish.

openssl req  -nodes -new -x509  -keyout private.key -out public.cert

In the Service Provider Data section enter the public certificate and the private key in the corresponding text blocks. It should look like this:

Identity Provider Data

In the Identity Provider section enter in your Keycloak settings. The format is

https://<your domain name here>/auth/realms/<your realm name here>

The last text entry is the RSA public key from your Keycloak server. This is found in the Master realm --> Realm Settings --> Keys https:///auth/admin/master/console/#/realms/master/keys

Attribute mapping

I did not do any special Attribute mapping.

Security settings

Check the boxes to make it look like this:

Export XML

With the Nextcloud configuration complete you should have a gray indication "Metadata valid" next to the "Download metadata XML" button. Download the XML file so that you can import it into Keycloak.

Keycloak configuration

In Keycloak create a new client and choose to import a file. Upload the previously downloaded medata.xml and click ok.

Then go to the Mappers tab. If you do not need the role mapping, just delete it. I don't use it yet but I might look at this again in the future.

Then create a new Mappers:

Username
name: username
mapper type: user property
property: username
SAML attribute name: username
SAML attribute NameFormat: basic

Email
name: email
mapper type: user property
property: email
SAML attribute name: email
SAML attribute NameFormat: basic

Roles
name: Roles
mapper type: Role List
Role attribute name: Roles
Friendly name: roles
SAML Attribute NameFormat: Basic
Single Role Attribute: On

After the mappers are created and the client is saved, all Keycloak configuration is done. Open a new browser/Private Window and try to login to your Nextcloud account. If it doesn't work, check out the Nextcloud log file and the Keycloak log to determine where the breakdown is happening.

B&R Automation Studio and PVI Windows Services

2021-12-08T06:22:00-06:00

1. ADVISORY INFORMATION

Product: B&R Automation Net/PVI
Vendor URL: https://www.br-automation.com/en-us/products/software/automation-netpvi/
Vulnerability #1: CVE-2020-24681 Automation Studio incorrect permission assignments for services. CVSS v3 Base Score: 8.2 (High)
Vulnerability #2: CVE-2020-24681 Net/PVI incorrect permission assignments for services. CVSS v3 Base Score: 8.2 (High)
Vulnerability #3: CVE-2020-24682 Automation Studio unquoted service path vulnerabilities. CVSS v3 Base Score: 7.2 (High)
Vulnerability #4: CVE-2020-24682 PVI Multiple unquoted service path vulnerabilities CVSS v3 Base Score: 7.2 (High)
Date found: 2020-09-28

2. AUTHOR

This vulnerability was discovered and researched by Andrew Hofmans
Public key: pubkey

3. VERSIONS AFFECTED

Vulnerabilities were found and tested on the most recently released version available at the time (PVI_4.8.2.27 07/27/2020).

4. INTRODUCTION

Our slogan is our mission. The pursuit of Perfection in Automation has inspired and guided B&R for over 40 years. To us, perfection means more than developing the best solutions in industrial automation. It also means developing the best relationships – with our customers and partners as well as our employees and suppliers.
(from the vendor's homepage)

5. VULNERABILITY DETAILS

A completely default PVI Automation NET installation installs a service named "BrDiskImageSvcx". The service is ran as "NT AUTHORITY\SYSTEM" and it is set to start Manually. The permissions for the service are set to "NT AUTHORITY\Authenticated Users". It is possible for any user, to include a non-administrative user account, to change the service bin path to an arbitrary executable and then start the service granting code execution as "NT AUTHORITY\SYSTEM". This is a local privilege escalation attack vector. (SC CONFIG BrDiskImageSvcx binPath= "C:\Users\NonAdmin\Downloads\foo.exe") (SC start BrDiskImageSvcx)
During PVI Automation NET installation, if the "Install PVI Manager as a service" component is selected, a service named "PviManSvcx" is created. The service is ran as "NT AUTHORITY\SYSTEM" and it is set to start Manually. The permissions for the service are set to "NT AUTHORITY\Authenticated Users". It is possible for any user, to include a non-administrative user account, to change the service bin path to an arbitrary executable and then start the service granting code execution as "NT AUTHORITY\SYSTEM". This is a local privilege escalation attack vector. (SC CONFIG PviManSvcx binPath= "C:\Users\NonAdmin\Downloads\foo.exe") (SC start PviManSvcx)
When the BrDiskImageSvcx service is created during the PVI installation, the "Path to executable" is not quoted. If the default installation directory, C:\BRAutomation, is used it isn't exploitable, but if the installation path is changed to any folder with a space (e.g. "Program Files", "Program Files (x86)") this leads to an unquoted service path vulnerability. A user with the necessary permissions to write to the root folder where the folder that has the space is located can get code execution as "NT AUTHORITY\SYSTEM" when the service is started. (C:\Program.exe) (SC start BrDiskImageSvcx)
If the "Install PVI Manager as a service" component is selected, the PviManSvcx service is created during the PVI installation with the "Path to executable" not quoted. If the default installation directory, C:\BRAutomation, is used it isn't exploitable, but if the installation path is changed to any folder with a space (e.g. "Program Files", "Program Files (x86)") this leads to an unquoted service path vulnerability. A user with the necessary permissions to write to the root folder where the folder that has the space is located can get code execution as "NT AUTHORITY\SYSTEM" when the service is started. (C:\Program.exe) (SC start PviManSvcx)

6. RISK

To successfully exploit this vulnerability an attacker must have access to the PC with the PVI Automation NET software installed. Any user in the "NT AUTHORITY\Authenticated Users", to include non-administrative users, can modify the service.
To successfully exploit this vulnerability an attacker must have access to the PC with the PVI Automation NET software installed and the "Install PVI Manager as a service" component was selected during program installation. Any user in the "NT AUTHORITY\Authenticated Users", to include non-administrative users, can modify the service.
To successfully exploit this vulnerability an attacker must have access to the PC with the PVI Automation NET software installed in a non-default installation directory, the installation directory has a space in the name, and the attacker has the necessary privileges to write to the same parent directory as the installation folder.
To successfully exploit this vulnerability an attacker must have access to the PC with the PVI Automation NET software installed in a non-default installation directory, the installation directory has a space in the name, the "Install PVI Manager as a service" component was selected during program installation, and the attacker has the necessary privileges to write to the same parent directory as the installation folder.

All the above vulnerabilities result in total device compromise with code execution as "NT AUTHORITY\SYSTEM".

7. Timeline

2020-09-28: Reported to B&R.

...A lot of back and forth, delays, and missed deadlines...

2021-11-30: B&R publishes security advisory. Also available here.

DLL Hijacking Vulnerability in Automation Studio

2021-12-08T06:22:00-06:00

1. ADVISORY INFORMATION

Product: B&R Automation Net/PVI
Vendor URL: https://www.br-automation.com/en-us/products/software/automation-netpvi/
Vulnerability #1: CVE-2021-22280 DLL Hijacking Vulnerability in Automation Studio. CVSS v3 Base Score: 7.2 (High)
Vulnerability #2: Improper Input Validation CWE-20
Date found: 2020-10-09

2. AUTHOR

This vulnerability was discovered and researched by Andrew Hofmans
Public key: pubkey

3. VERSIONS AFFECTED

Vulnerabilities were found and tested on the most recently released version available at the time (PVI_4.8.2.27 07/27/2020).

4. VULNERABILITY DETAILS

When the PVI Manager process is started, it attempts to load DLL resources that are not installed in the local directory. When the DLLs are not found, the process starts searching the directories in the System Path. By placing a malicious DLL in one of the directories found in the System Path, it is possible to get code execution as the user and integrity level as the owner of PVIMan.exe.
- PVI Manager is automatically started when the PVI Monitor application is run. If the "PviManSvcx" is installed the service will start. This causes the malicious DLLs to be loaded as "NT AUTHORITY\SYSTEM".
- If "PviManSvcx" is not installed, the process will be owned and executed by the user that started it.
  - Exploit: Create a malicious DLL (msfvenom -a x64 -f dll -p windows/x64/exec CMD="C:\windows\system32\notepad.exe" -o BrAdi.dll)
    Place DLL in a directory in the System PATH.
    Start the "PviManSvcx" service or run the PVI Manager application.
    DLL loaded and code is executed.
  - Vulnerable DLLs (there may be more) BrAdi.dll
    PviLcMod.dll
    BrLcMod.dll
    IB10E64.dll
    IB97U64.dll
    IBUSB64.dll
PVI Manager loaded DLLs that were completely unsigned, much less not signed by B&R's digital signature.

5. RISK

To successfully exploit this vulnerability an attacker must have local access to the PC with the PVI Automation NET software installed. The attacker must have the necessary privileges to write to a directory in the System PATH. This allows arbitrary DLLs to be loaded resulting in unexpected code execution. If the process is started by starting the service "PviManSvcx", it is run by the SYSTEM account resulting in total device compromise.
No signatures are checked prior to the DLLs being loaded. This allows arbitrary and untrusted code to be loaded into a process owned by SYSTEM.

6. Timeline

2020-10-11: Reported to B&R.

...A lot of back and forth, delays, and missed deadlines...

2021-10-29: B&R publishes security advisory. Also available here.

Mirrored Ports with Open vSwitch

2020-04-15T16:00:00-05:00

After adding some new SSD storage to my home server I had purged a lot of the older VMs that I didn't use frequently in my previous 'Spring Cleaning' effort to make space. One of the VMs that I had purged but now wanted to get back online and running was a Security Onion VM. I won't go into much detail here about SecOnion, there is a lot of good documentation and tutorials already out there on how to set it up. Security Onion requires at least two network interfaces; one for administrative control, and at least one more that ingests the network traffic. The second+ NIC will be set to promiscuous mode in the OS and the actual hardware switch port should be set to SPAN or MIRROR (whatever your manufacturer calls it).

Switch Configuration

For my home network I'm using a variety of Ubiquiti devices. I have an EdgeRouter Lite -> Unifi Switch 8 POE-60W -> UniFi AC Pro AP. I can't say enough how impressed I am overall in the performance and quality of the devices. I found them intuitive to configure and offer the enterprise class features that I wanted for my lab. To configure the switch, I open the devices settings in my self-hosted unifi controller and select the port I want to do the mirroring on. I'll be making port 4 mirror port 1 which is my router -> switch connection. Switch Port 4 is already connected to a seperate network card on the Proxmox server. Thats all we need to do to configure the switch for mirroring. Easy!

Proxmox Configuration

The Proxmox configuration was more difficult to figure out (hence this article) and is only partly doable using the web GUI. I'll present both here, but I think the command line version is much faster since you must use it anyways.

GUI

To start out, open the Proxmox web interface and navigate to the node's network page. Create a new OVS Bridge and in the "Bridge Ports:" section add the NIC that has the mirrored traffic. Click create and then reboot the server for the network changes to be created. From here you will need a shell on the server.

Command Line

As a root user open your /etc/network/interfaces in your favorite editor. Create a new interface and set the type to OVSBridge and add the physical interface as an OVSPort. It should look similar to the below:

auto vmbr2
iface vmbr2 inet manual
    ovs_type OVSBridge
    ovs_ports enp7s0f0

Restart the networking service

service networking restart

Now that the bridge is available in Proxmox, you have to add your Security Onions interface to it. Proxmox creates a tap interface for the VMs that it hosts so you can find your VMs interface with a simple "ifconfig". The naming convention is tap-VM number-i-interface number. For mine it was tap10000i1.

tap10000i1: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST>  mtu 1500
    ether 16:b3:72:aa:39:a9  txqueuelen 1000  (Ethernet)
    RX packets 1  bytes 90 (90.0 B)
    RX errors 0  dropped 0  overruns 0  frame 0
    TX packets 225617  bytes 203347587 (193.9 MiB)
    TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

To finally add the interface, we need to use the instructions available on the the Open vSwitch website. My command was the following:

ovs-vsctl -- --id=@p get port tap10000i1 \ 
-- --id=@m create mirror name=sopan1 select-all=true output-port=@p \
-- set bridge vmbr2 mirrors=@m

An important note to make, is that because you need the tap the VM is using, the Security Onion instance MUST be running prior to running the OVS command. Later, this created some confusion for me as this command does NOT survive a reboot and had to be performed again, and then again when I actually started the VM. I solved that problem by discovering a similar article by "vext" in his Proxmox / Security Onion write up. I ended up ripping off his idea and used his script and added it to my crontab for an easy, repeatable, and reliable reboot.

Access Git repo over SSH behind jumpbox

2020-01-13T16:00:00-06:00

I recently ran accross the situation where I wanted to push changes to a repo that was stored on my personal Gitlab server. The issue was that I had recently restructured my homelab and the only way to get to the gitlab server via SSH was through a jumpbox. These were the steps I took to be able to use the normal git workflow while still keeping my network design / security choices intact.

On dev machine

First we will create a new SSH key for the hop from dev machine --> jumpbox.

ssh-keygen -t ed25519 -C "gitlab multihop" -f ~/.ssh/multihop.ed25519 -N ""

This creates a new SSH key (ed25519 type) called multihop.ed25519 with no passphrase and stored in our profile's .ssh directory.

Copy it to the jumpbox

ssh-copy-id -i ~/.ssh/multihop.ed25519 username@jumpbox

Now edit (or create) your profile's SSH config file and add in the host parameters.

vi ~/.ssh/config

Host jumpbox
  User jumpboxusernamehere
  Port 22
  IdentityFile ~/.ssh/multihop.ed25519



Host gitlab
  Hostname gitlab.ip.or.hostname
  Port 22
  ProxyJump jumpbox.ip.or.hostname
  User gitlabboxusername
  IdentityFile ~/.ssh/multihop_ed25519

Now you can use the standard git commands and connect to your gitlab instance through an SSH jumphost.

Install RDP on Debian 10 (Buster)

2020-01-09T16:00:00-06:00

Start off with a freshly updated system and install the requirements:

apt update
apt upgrade
apt install -y xrdp xfce4 xfce4-terminal gnome-icon-theme-full tango-icon-theme

Make xfce4 our default xsession:

echo xfce4-session >~/.xsession

Make xfce4 the default for by editting the following file: :::bash nano /etc/xrdp/startwm.sh

Replace what is in there with this:

#!/bin/sh

if [ -r /etc/default/locale ]; then
  . /etc/default/locale
  export LANG LANGUAGE
fi

startxfce4

Restart xrdp for changes to be made effective.

service xrdp restart

GPG random-entropy

2019-05-16T16:07:00-05:00

This is was the workaround that I had to do to get gpg keys to be generated on a kvm debian machine. After doing it several times and having to hunt to figure out how I got it working the first time I decided I should write it down.....

sudo apt-get install rng-tools
sudo rngd -r /dev/urandom

Road Warrior VPN

2019-05-06T17:22:00-05:00

This is a dead simple way to get an OpenVPN server up and going with individual client certificates.

wget https://git.io/vpn -O openvpn-install.sh && bash openvpn-install.sh

#!/bin/bash
#
# https://github.com/Nyr/openvpn-install
#
# Copyright (c) 2013 Nyr. Released under the MIT License.


# Detect Debian users running the script with "sh" instead of bash
if readlink /proc/$$/exe | grep -q "dash"; then
    echo 'This installer needs to be run with "bash", not "sh".'
    exit
fi

# Discard stdin. Needed when running from an one-liner which includes a newline
read -N 999999 -t 0.001

# Detect OpenVZ 6
if [[ $(uname -r | cut -d "." -f 1) -eq 2 ]]; then
    echo "The system is running an old kernel, which is incompatible with this installer."
    exit
fi

# Detect OS
# $os_version variables aren't always in use, but are kept here for convenience
if grep -qs "ubuntu" /etc/os-release; then
    os="ubuntu"
    os_version=$(grep 'VERSION_ID' /etc/os-release | cut -d '"' -f 2 | tr -d '.')
    group_name="nogroup"
elif [[ -e /etc/debian_version ]]; then
    os="debian"
    os_version=$(grep -oE '[0-9]+' /etc/debian_version | head -1)
    group_name="nogroup"
elif [[ -e /etc/almalinux-release || -e /etc/rocky-release || -e /etc/centos-release ]]; then
    os="centos"
    os_version=$(grep -shoE '[0-9]+' /etc/almalinux-release /etc/rocky-release /etc/centos-release | head -1)
    group_name="nobody"
elif [[ -e /etc/fedora-release ]]; then
    os="fedora"
    os_version=$(grep -oE '[0-9]+' /etc/fedora-release | head -1)
    group_name="nobody"
else
    echo "This installer seems to be running on an unsupported distribution.
Supported distros are Ubuntu, Debian, AlmaLinux, Rocky Linux, CentOS and Fedora."
    exit
fi

if [[ "$os" == "ubuntu" && "$os_version" -lt 1804 ]]; then
    echo "Ubuntu 18.04 or higher is required to use this installer.
This version of Ubuntu is too old and unsupported."
    exit
fi

if [[ "$os" == "debian" && "$os_version" -lt 9 ]]; then
    echo "Debian 9 or higher is required to use this installer.
This version of Debian is too old and unsupported."
    exit
fi

if [[ "$os" == "centos" && "$os_version" -lt 7 ]]; then
    echo "CentOS 7 or higher is required to use this installer.
This version of CentOS is too old and unsupported."
    exit
fi

# Detect environments where $PATH does not include the sbin directories
if ! grep -q sbin <<< "$PATH"; then
    echo '$PATH does not include sbin. Try using "su -" instead of "su".'
    exit
fi

if [[ "$EUID" -ne 0 ]]; then
    echo "This installer needs to be run with superuser privileges."
    exit
fi

if [[ ! -e /dev/net/tun ]] || ! ( exec 7<>/dev/net/tun ) 2>/dev/null; then
    echo "The system does not have the TUN device available.
TUN needs to be enabled before running this installer."
    exit
fi

new_client () {
    # Generates the custom client.ovpn
    {
    cat /etc/openvpn/server/client-common.txt
    echo "<ca>"
    cat /etc/openvpn/server/easy-rsa/pki/ca.crt
    echo "</ca>"
    echo "<cert>"
    sed -ne '/BEGIN CERTIFICATE/,$ p' /etc/openvpn/server/easy-rsa/pki/issued/"$client".crt
    echo "</cert>"
    echo "<key>"
    cat /etc/openvpn/server/easy-rsa/pki/private/"$client".key
    echo "</key>"
    echo "<tls-crypt>"
    sed -ne '/BEGIN OpenVPN Static key/,$ p' /etc/openvpn/server/tc.key
    echo "</tls-crypt>"
    } > ~/"$client".ovpn
}

if [[ ! -e /etc/openvpn/server/server.conf ]]; then
    # Detect some Debian minimal setups where neither wget nor curl are installed
    if ! hash wget 2>/dev/null && ! hash curl 2>/dev/null; then
        echo "Wget is required to use this installer."
        read -n1 -r -p "Press any key to install Wget and continue..."
        apt-get update
        apt-get install -y wget
    fi
    clear
    echo 'Welcome to this OpenVPN road warrior installer!'
    # If system has a single IPv4, it is selected automatically. Else, ask the user
    if [[ $(ip -4 addr | grep inet | grep -vEc '127(\.[0-9]{1,3}){3}') -eq 1 ]]; then
        ip=$(ip -4 addr | grep inet | grep -vE '127(\.[0-9]{1,3}){3}' | cut -d '/' -f 1 | grep -oE '[0-9]{1,3}(\.[0-9]{1,3}){3}')
    else
        number_of_ip=$(ip -4 addr | grep inet | grep -vEc '127(\.[0-9]{1,3}){3}')
        echo
        echo "Which IPv4 address should be used?"
        ip -4 addr | grep inet | grep -vE '127(\.[0-9]{1,3}){3}' | cut -d '/' -f 1 | grep -oE '[0-9]{1,3}(\.[0-9]{1,3}){3}' | nl -s ') '
        read -p "IPv4 address [1]: " ip_number
        until [[ -z "$ip_number" || "$ip_number" =~ ^[0-9]+$ && "$ip_number" -le "$number_of_ip" ]]; do
            echo "$ip_number: invalid selection."
            read -p "IPv4 address [1]: " ip_number
        done
        [[ -z "$ip_number" ]] && ip_number="1"
        ip=$(ip -4 addr | grep inet | grep -vE '127(\.[0-9]{1,3}){3}' | cut -d '/' -f 1 | grep -oE '[0-9]{1,3}(\.[0-9]{1,3}){3}' | sed -n "$ip_number"p)
    fi
    # If $ip is a private IP address, the server must be behind NAT
    if echo "$ip" | grep -qE '^(10\.|172\.1[6789]\.|172\.2[0-9]\.|172\.3[01]\.|192\.168)'; then
        echo
        echo "This server is behind NAT. What is the public IPv4 address or hostname?"
        # Get public IP and sanitize with grep
        get_public_ip=$(grep -m 1 -oE '^[0-9]{1,3}(\.[0-9]{1,3}){3}$' <<< "$(wget -T 10 -t 1 -4qO- "http://ip1.dynupdate.no-ip.com/" || curl -m 10 -4Ls "http://ip1.dynupdate.no-ip.com/")")
        read -p "Public IPv4 address / hostname [$get_public_ip]: " public_ip
        # If the checkip service is unavailable and user didn't provide input, ask again
        until [[ -n "$get_public_ip" || -n "$public_ip" ]]; do
            echo "Invalid input."
            read -p "Public IPv4 address / hostname: " public_ip
        done
        [[ -z "$public_ip" ]] && public_ip="$get_public_ip"
    fi
    # If system has a single IPv6, it is selected automatically
    if [[ $(ip -6 addr | grep -c 'inet6 [23]') -eq 1 ]]; then
        ip6=$(ip -6 addr | grep 'inet6 [23]' | cut -d '/' -f 1 | grep -oE '([0-9a-fA-F]{0,4}:){1,7}[0-9a-fA-F]{0,4}')
    fi
    # If system has multiple IPv6, ask the user to select one
    if [[ $(ip -6 addr | grep -c 'inet6 [23]') -gt 1 ]]; then
        number_of_ip6=$(ip -6 addr | grep -c 'inet6 [23]')
        echo
        echo "Which IPv6 address should be used?"
        ip -6 addr | grep 'inet6 [23]' | cut -d '/' -f 1 | grep -oE '([0-9a-fA-F]{0,4}:){1,7}[0-9a-fA-F]{0,4}' | nl -s ') '
        read -p "IPv6 address [1]: " ip6_number
        until [[ -z "$ip6_number" || "$ip6_number" =~ ^[0-9]+$ && "$ip6_number" -le "$number_of_ip6" ]]; do
            echo "$ip6_number: invalid selection."
            read -p "IPv6 address [1]: " ip6_number
        done
        [[ -z "$ip6_number" ]] && ip6_number="1"
        ip6=$(ip -6 addr | grep 'inet6 [23]' | cut -d '/' -f 1 | grep -oE '([0-9a-fA-F]{0,4}:){1,7}[0-9a-fA-F]{0,4}' | sed -n "$ip6_number"p)
    fi
    echo
    echo "Which protocol should OpenVPN use?"
    echo "   1) UDP (recommended)"
    echo "   2) TCP"
    read -p "Protocol [1]: " protocol
    until [[ -z "$protocol" || "$protocol" =~ ^[12]$ ]]; do
        echo "$protocol: invalid selection."
        read -p "Protocol [1]: " protocol
    done
    case "$protocol" in
        1|"") 
        protocol=udp
        ;;
        2) 
        protocol=tcp
        ;;
    esac
    echo
    echo "What port should OpenVPN listen to?"
    read -p "Port [1194]: " port
    until [[ -z "$port" || "$port" =~ ^[0-9]+$ && "$port" -le 65535 ]]; do
        echo "$port: invalid port."
        read -p "Port [1194]: " port
    done
    [[ -z "$port" ]] && port="1194"
    echo
    echo "Select a DNS server for the clients:"
    echo "   1) Current system resolvers"
    echo "   2) Google"
    echo "   3) 1.1.1.1"
    echo "   4) OpenDNS"
    echo "   5) Quad9"
    echo "   6) AdGuard"
    read -p "DNS server [1]: " dns
    until [[ -z "$dns" || "$dns" =~ ^[1-6]$ ]]; do
        echo "$dns: invalid selection."
        read -p "DNS server [1]: " dns
    done
    echo
    echo "Enter a name for the first client:"
    read -p "Name [client]: " unsanitized_client
    # Allow a limited set of characters to avoid conflicts
    client=$(sed 's/[^0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ_-]/_/g' <<< "$unsanitized_client")
    [[ -z "$client" ]] && client="client"
    echo
    echo "OpenVPN installation is ready to begin."
    # Install a firewall if firewalld or iptables are not already available
    if ! systemctl is-active --quiet firewalld.service && ! hash iptables 2>/dev/null; then
        if [[ "$os" == "centos" || "$os" == "fedora" ]]; then
            firewall="firewalld"
            # We don't want to silently enable firewalld, so we give a subtle warning
            # If the user continues, firewalld will be installed and enabled during setup
            echo "firewalld, which is required to manage routing tables, will also be installed."
        elif [[ "$os" == "debian" || "$os" == "ubuntu" ]]; then
            # iptables is way less invasive than firewalld so no warning is given
            firewall="iptables"
        fi
    fi
    read -n1 -r -p "Press any key to continue..."
    # If running inside a container, disable LimitNPROC to prevent conflicts
    if systemd-detect-virt -cq; then
        mkdir /etc/systemd/system/openvpn-server@server.service.d/ 2>/dev/null
        echo "[Service]
LimitNPROC=infinity" > /etc/systemd/system/openvpn-server@server.service.d/disable-limitnproc.conf
    fi
    if [[ "$os" = "debian" || "$os" = "ubuntu" ]]; then
        apt-get update
        apt-get install -y --no-install-recommends openvpn openssl ca-certificates $firewall
    elif [[ "$os" = "centos" ]]; then
        yum install -y epel-release
        yum install -y openvpn openssl ca-certificates tar $firewall
    else
        # Else, OS must be Fedora
        dnf install -y openvpn openssl ca-certificates tar $firewall
    fi
    # If firewalld was just installed, enable it
    if [[ "$firewall" == "firewalld" ]]; then
        systemctl enable --now firewalld.service
    fi
    # Get easy-rsa
    easy_rsa_url='https://github.com/OpenVPN/easy-rsa/releases/download/v3.1.2/EasyRSA-3.1.2.tgz'
    mkdir -p /etc/openvpn/server/easy-rsa/
    { wget -qO- "$easy_rsa_url" 2>/dev/null || curl -sL "$easy_rsa_url" ; } | tar xz -C /etc/openvpn/server/easy-rsa/ --strip-components 1
    chown -R root:root /etc/openvpn/server/easy-rsa/
    cd /etc/openvpn/server/easy-rsa/
    # Create the PKI, set up the CA and the server and client certificates
    ./easyrsa --batch init-pki
    ./easyrsa --batch build-ca nopass
    ./easyrsa --batch --days=3650 build-server-full server nopass
    ./easyrsa --batch --days=3650 build-client-full "$client" nopass
    ./easyrsa --batch --days=3650 gen-crl
    # Move the stuff we need
    cp pki/ca.crt pki/private/ca.key pki/issued/server.crt pki/private/server.key pki/crl.pem /etc/openvpn/server
    # CRL is read with each client connection, while OpenVPN is dropped to nobody
    chown nobody:"$group_name" /etc/openvpn/server/crl.pem
    # Without +x in the directory, OpenVPN can't run a stat() on the CRL file
    chmod o+x /etc/openvpn/server/
    # Generate key for tls-crypt
    openvpn --genkey --secret /etc/openvpn/server/tc.key
    # Create the DH parameters file using the predefined ffdhe2048 group
    echo '-----BEGIN DH PARAMETERS-----
MIIBCAKCAQEA//////////+t+FRYortKmq/cViAnPTzx2LnFg84tNpWp4TZBFGQz
+8yTnc4kmz75fS/jY2MMddj2gbICrsRhetPfHtXV/WVhJDP1H18GbtCFY2VVPe0a
87VXE15/V8k1mE8McODmi3fipona8+/och3xWKE2rec1MKzKT0g6eXq8CrGCsyT7
YdEIqUuyyOP7uWrat2DX9GgdT0Kj3jlN9K5W7edjcrsZCwenyO4KbXCeAvzhzffi
7MA0BM0oNC9hkXL+nOmFg/+OTxIy7vKBg8P+OxtMb61zO7X8vC7CIAXFjvGDfRaD
ssbzSibBsu/6iGtCOGEoXJf//////////wIBAg==
-----END DH PARAMETERS-----' > /etc/openvpn/server/dh.pem
    # Generate server.conf
    echo "local $ip
port $port
proto $protocol
dev tun
ca ca.crt
cert server.crt
key server.key
dh dh.pem
auth SHA512
tls-crypt tc.key
topology subnet
server 10.8.0.0 255.255.255.0" > /etc/openvpn/server/server.conf
    # IPv6
    if [[ -z "$ip6" ]]; then
        echo 'push "redirect-gateway def1 bypass-dhcp"' >> /etc/openvpn/server/server.conf
    else
        echo 'server-ipv6 fddd:1194:1194:1194::/64' >> /etc/openvpn/server/server.conf
        echo 'push "redirect-gateway def1 ipv6 bypass-dhcp"' >> /etc/openvpn/server/server.conf
    fi
    echo 'ifconfig-pool-persist ipp.txt' >> /etc/openvpn/server/server.conf
    # DNS
    case "$dns" in
        1|"")
            # Locate the proper resolv.conf
            # Needed for systems running systemd-resolved
            if grep '^nameserver' "/etc/resolv.conf" | grep -qv '127.0.0.53' ; then
                resolv_conf="/etc/resolv.conf"
            else
                resolv_conf="/run/systemd/resolve/resolv.conf"
            fi
            # Obtain the resolvers from resolv.conf and use them for OpenVPN
            grep -v '^#\|^;' "$resolv_conf" | grep '^nameserver' | grep -v '127.0.0.53' | grep -oE '[0-9]{1,3}(\.[0-9]{1,3}){3}' | while read line; do
                echo "push \"dhcp-option DNS $line\"" >> /etc/openvpn/server/server.conf
            done
        ;;
        2)
            echo 'push "dhcp-option DNS 8.8.8.8"' >> /etc/openvpn/server/server.conf
            echo 'push "dhcp-option DNS 8.8.4.4"' >> /etc/openvpn/server/server.conf
        ;;
        3)
            echo 'push "dhcp-option DNS 1.1.1.1"' >> /etc/openvpn/server/server.conf
            echo 'push "dhcp-option DNS 1.0.0.1"' >> /etc/openvpn/server/server.conf
        ;;
        4)
            echo 'push "dhcp-option DNS 208.67.222.222"' >> /etc/openvpn/server/server.conf
            echo 'push "dhcp-option DNS 208.67.220.220"' >> /etc/openvpn/server/server.conf
        ;;
        5)
            echo 'push "dhcp-option DNS 9.9.9.9"' >> /etc/openvpn/server/server.conf
            echo 'push "dhcp-option DNS 149.112.112.112"' >> /etc/openvpn/server/server.conf
        ;;
        6)
            echo 'push "dhcp-option DNS 94.140.14.14"' >> /etc/openvpn/server/server.conf
            echo 'push "dhcp-option DNS 94.140.15.15"' >> /etc/openvpn/server/server.conf
        ;;
    esac
    echo 'push "block-outside-dns"' >> /etc/openvpn/server/server.conf
    echo "keepalive 10 120
cipher AES-256-CBC
user nobody
group $group_name
persist-key
persist-tun
verb 3
crl-verify crl.pem" >> /etc/openvpn/server/server.conf
    if [[ "$protocol" = "udp" ]]; then
        echo "explicit-exit-notify" >> /etc/openvpn/server/server.conf
    fi
    # Enable net.ipv4.ip_forward for the system
    echo 'net.ipv4.ip_forward=1' > /etc/sysctl.d/99-openvpn-forward.conf
    # Enable without waiting for a reboot or service restart
    echo 1 > /proc/sys/net/ipv4/ip_forward
    if [[ -n "$ip6" ]]; then
        # Enable net.ipv6.conf.all.forwarding for the system
        echo "net.ipv6.conf.all.forwarding=1" >> /etc/sysctl.d/99-openvpn-forward.conf
        # Enable without waiting for a reboot or service restart
        echo 1 > /proc/sys/net/ipv6/conf/all/forwarding
    fi
    if systemctl is-active --quiet firewalld.service; then
        # Using both permanent and not permanent rules to avoid a firewalld
        # reload.
        # We don't use --add-service=openvpn because that would only work with
        # the default port and protocol.
        firewall-cmd --add-port="$port"/"$protocol"
        firewall-cmd --zone=trusted --add-source=10.8.0.0/24
        firewall-cmd --permanent --add-port="$port"/"$protocol"
        firewall-cmd --permanent --zone=trusted --add-source=10.8.0.0/24
        # Set NAT for the VPN subnet
        firewall-cmd --direct --add-rule ipv4 nat POSTROUTING 0 -s 10.8.0.0/24 ! -d 10.8.0.0/24 -j SNAT --to "$ip"
        firewall-cmd --permanent --direct --add-rule ipv4 nat POSTROUTING 0 -s 10.8.0.0/24 ! -d 10.8.0.0/24 -j SNAT --to "$ip"
        if [[ -n "$ip6" ]]; then
            firewall-cmd --zone=trusted --add-source=fddd:1194:1194:1194::/64
            firewall-cmd --permanent --zone=trusted --add-source=fddd:1194:1194:1194::/64
            firewall-cmd --direct --add-rule ipv6 nat POSTROUTING 0 -s fddd:1194:1194:1194::/64 ! -d fddd:1194:1194:1194::/64 -j SNAT --to "$ip6"
            firewall-cmd --permanent --direct --add-rule ipv6 nat POSTROUTING 0 -s fddd:1194:1194:1194::/64 ! -d fddd:1194:1194:1194::/64 -j SNAT --to "$ip6"
        fi
    else
        # Create a service to set up persistent iptables rules
        iptables_path=$(command -v iptables)
        ip6tables_path=$(command -v ip6tables)
        # nf_tables is not available as standard in OVZ kernels. So use iptables-legacy
        # if we are in OVZ, with a nf_tables backend and iptables-legacy is available.
        if [[ $(systemd-detect-virt) == "openvz" ]] && readlink -f "$(command -v iptables)" | grep -q "nft" && hash iptables-legacy 2>/dev/null; then
            iptables_path=$(command -v iptables-legacy)
            ip6tables_path=$(command -v ip6tables-legacy)
        fi
        echo "[Unit]
Before=network.target
[Service]
Type=oneshot
ExecStart=$iptables_path -t nat -A POSTROUTING -s 10.8.0.0/24 ! -d 10.8.0.0/24 -j SNAT --to $ip
ExecStart=$iptables_path -I INPUT -p $protocol --dport $port -j ACCEPT
ExecStart=$iptables_path -I FORWARD -s 10.8.0.0/24 -j ACCEPT
ExecStart=$iptables_path -I FORWARD -m state --state RELATED,ESTABLISHED -j ACCEPT
ExecStop=$iptables_path -t nat -D POSTROUTING -s 10.8.0.0/24 ! -d 10.8.0.0/24 -j SNAT --to $ip
ExecStop=$iptables_path -D INPUT -p $protocol --dport $port -j ACCEPT
ExecStop=$iptables_path -D FORWARD -s 10.8.0.0/24 -j ACCEPT
ExecStop=$iptables_path -D FORWARD -m state --state RELATED,ESTABLISHED -j ACCEPT" > /etc/systemd/system/openvpn-iptables.service
        if [[ -n "$ip6" ]]; then
            echo "ExecStart=$ip6tables_path -t nat -A POSTROUTING -s fddd:1194:1194:1194::/64 ! -d fddd:1194:1194:1194::/64 -j SNAT --to $ip6
ExecStart=$ip6tables_path -I FORWARD -s fddd:1194:1194:1194::/64 -j ACCEPT
ExecStart=$ip6tables_path -I FORWARD -m state --state RELATED,ESTABLISHED -j ACCEPT
ExecStop=$ip6tables_path -t nat -D POSTROUTING -s fddd:1194:1194:1194::/64 ! -d fddd:1194:1194:1194::/64 -j SNAT --to $ip6
ExecStop=$ip6tables_path -D FORWARD -s fddd:1194:1194:1194::/64 -j ACCEPT
ExecStop=$ip6tables_path -D FORWARD -m state --state RELATED,ESTABLISHED -j ACCEPT" >> /etc/systemd/system/openvpn-iptables.service
        fi
        echo "RemainAfterExit=yes
[Install]
WantedBy=multi-user.target" >> /etc/systemd/system/openvpn-iptables.service
        systemctl enable --now openvpn-iptables.service
    fi
    # If SELinux is enabled and a custom port was selected, we need this
    if sestatus 2>/dev/null | grep "Current mode" | grep -q "enforcing" && [[ "$port" != 1194 ]]; then
        # Install semanage if not already present
        if ! hash semanage 2>/dev/null; then
            if [[ "$os_version" -eq 7 ]]; then
                # Centos 7
                yum install -y policycoreutils-python
            else
                # CentOS 8 or Fedora
                dnf install -y policycoreutils-python-utils
            fi
        fi
        semanage port -a -t openvpn_port_t -p "$protocol" "$port"
    fi
    # If the server is behind NAT, use the correct IP address
    [[ -n "$public_ip" ]] && ip="$public_ip"
    # client-common.txt is created so we have a template to add further users later
    echo "client
dev tun
proto $protocol
remote $ip $port
resolv-retry infinite
nobind
persist-key
persist-tun
remote-cert-tls server
auth SHA512
cipher AES-256-CBC
ignore-unknown-option block-outside-dns
verb 3" > /etc/openvpn/server/client-common.txt
    # Enable and start the OpenVPN service
    systemctl enable --now openvpn-server@server.service
    # Generates the custom client.ovpn
    new_client
    echo
    echo "Finished!"
    echo
    echo "The client configuration is available in:" ~/"$client.ovpn"
    echo "New clients can be added by running this script again."
else
    clear
    echo "OpenVPN is already installed."
    echo
    echo "Select an option:"
    echo "   1) Add a new client"
    echo "   2) Revoke an existing client"
    echo "   3) Remove OpenVPN"
    echo "   4) Exit"
    read -p "Option: " option
    until [[ "$option" =~ ^[1-4]$ ]]; do
        echo "$option: invalid selection."
        read -p "Option: " option
    done
    case "$option" in
        1)
            echo
            echo "Provide a name for the client:"
            read -p "Name: " unsanitized_client
            client=$(sed 's/[^0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ_-]/_/g' <<< "$unsanitized_client")
            while [[ -z "$client" || -e /etc/openvpn/server/easy-rsa/pki/issued/"$client".crt ]]; do
                echo "$client: invalid name."
                read -p "Name: " unsanitized_client
                client=$(sed 's/[^0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ_-]/_/g' <<< "$unsanitized_client")
            done
            cd /etc/openvpn/server/easy-rsa/
            ./easyrsa --batch --days=3650 build-client-full "$client" nopass
            # Generates the custom client.ovpn
            new_client
            echo
            echo "$client added. Configuration available in:" ~/"$client.ovpn"
            exit
        ;;
        2)
            # This option could be documented a bit better and maybe even be simplified
            # ...but what can I say, I want some sleep too
            number_of_clients=$(tail -n +2 /etc/openvpn/server/easy-rsa/pki/index.txt | grep -c "^V")
            if [[ "$number_of_clients" = 0 ]]; then
                echo
                echo "There are no existing clients!"
                exit
            fi
            echo
            echo "Select the client to revoke:"
            tail -n +2 /etc/openvpn/server/easy-rsa/pki/index.txt | grep "^V" | cut -d '=' -f 2 | nl -s ') '
            read -p "Client: " client_number
            until [[ "$client_number" =~ ^[0-9]+$ && "$client_number" -le "$number_of_clients" ]]; do
                echo "$client_number: invalid selection."
                read -p "Client: " client_number
            done
            client=$(tail -n +2 /etc/openvpn/server/easy-rsa/pki/index.txt | grep "^V" | cut -d '=' -f 2 | sed -n "$client_number"p)
            echo
            read -p "Confirm $client revocation? [y/N]: " revoke
            until [[ "$revoke" =~ ^[yYnN]*$ ]]; do
                echo "$revoke: invalid selection."
                read -p "Confirm $client revocation? [y/N]: " revoke
            done
            if [[ "$revoke" =~ ^[yY]$ ]]; then
                cd /etc/openvpn/server/easy-rsa/
                ./easyrsa --batch revoke "$client"
                ./easyrsa --batch --days=3650 gen-crl
                rm -f /etc/openvpn/server/crl.pem
                cp /etc/openvpn/server/easy-rsa/pki/crl.pem /etc/openvpn/server/crl.pem
                # CRL is read with each client connection, when OpenVPN is dropped to nobody
                chown nobody:"$group_name" /etc/openvpn/server/crl.pem
                echo
                echo "$client revoked!"
            else
                echo
                echo "$client revocation aborted!"
            fi
            exit
        ;;
        3)
            echo
            read -p "Confirm OpenVPN removal? [y/N]: " remove
            until [[ "$remove" =~ ^[yYnN]*$ ]]; do
                echo "$remove: invalid selection."
                read -p "Confirm OpenVPN removal? [y/N]: " remove
            done
            if [[ "$remove" =~ ^[yY]$ ]]; then
                port=$(grep '^port ' /etc/openvpn/server/server.conf | cut -d " " -f 2)
                protocol=$(grep '^proto ' /etc/openvpn/server/server.conf | cut -d " " -f 2)
                if systemctl is-active --quiet firewalld.service; then
                    ip=$(firewall-cmd --direct --get-rules ipv4 nat POSTROUTING | grep '\-s 10.8.0.0/24 '"'"'!'"'"' -d 10.8.0.0/24' | grep -oE '[^ ]+$')
                    # Using both permanent and not permanent rules to avoid a firewalld reload.
                    firewall-cmd --remove-port="$port"/"$protocol"
                    firewall-cmd --zone=trusted --remove-source=10.8.0.0/24
                    firewall-cmd --permanent --remove-port="$port"/"$protocol"
                    firewall-cmd --permanent --zone=trusted --remove-source=10.8.0.0/24
                    firewall-cmd --direct --remove-rule ipv4 nat POSTROUTING 0 -s 10.8.0.0/24 ! -d 10.8.0.0/24 -j SNAT --to "$ip"
                    firewall-cmd --permanent --direct --remove-rule ipv4 nat POSTROUTING 0 -s 10.8.0.0/24 ! -d 10.8.0.0/24 -j SNAT --to "$ip"
                    if grep -qs "server-ipv6" /etc/openvpn/server/server.conf; then
                        ip6=$(firewall-cmd --direct --get-rules ipv6 nat POSTROUTING | grep '\-s fddd:1194:1194:1194::/64 '"'"'!'"'"' -d fddd:1194:1194:1194::/64' | grep -oE '[^ ]+$')
                        firewall-cmd --zone=trusted --remove-source=fddd:1194:1194:1194::/64
                        firewall-cmd --permanent --zone=trusted --remove-source=fddd:1194:1194:1194::/64
                        firewall-cmd --direct --remove-rule ipv6 nat POSTROUTING 0 -s fddd:1194:1194:1194::/64 ! -d fddd:1194:1194:1194::/64 -j SNAT --to "$ip6"
                        firewall-cmd --permanent --direct --remove-rule ipv6 nat POSTROUTING 0 -s fddd:1194:1194:1194::/64 ! -d fddd:1194:1194:1194::/64 -j SNAT --to "$ip6"
                    fi
                else
                    systemctl disable --now openvpn-iptables.service
                    rm -f /etc/systemd/system/openvpn-iptables.service
                fi
                if sestatus 2>/dev/null | grep "Current mode" | grep -q "enforcing" && [[ "$port" != 1194 ]]; then
                    semanage port -d -t openvpn_port_t -p "$protocol" "$port"
                fi
                systemctl disable --now openvpn-server@server.service
                rm -f /etc/systemd/system/openvpn-server@server.service.d/disable-limitnproc.conf
                rm -f /etc/sysctl.d/99-openvpn-forward.conf
                if [[ "$os" = "debian" || "$os" = "ubuntu" ]]; then
                    rm -rf /etc/openvpn/server
                    apt-get remove --purge -y openvpn
                else
                    # Else, OS must be CentOS or Fedora
                    yum remove -y openvpn
                    rm -rf /etc/openvpn/server
                fi
                echo
                echo "OpenVPN removed!"
            else
                echo
                echo "OpenVPN removal aborted!"
            fi
            exit
        ;;
        4)
            exit
        ;;
    esac
fi

Mesowx Real Time Weather Graphing

2017-04-23T16:00:00-05:00

To start off this tutorial it is assumed that you have Weewx completely set up on another server with the weather station properly connected and reporting. This tutorial will be using a Master / Slave MYSQL database that is configured like my previous blog entry.

The purpose adding Mesowx into the mix was not to add another layer of complication into the mix, but to be able to dynamically display a dataset and to have the ability to focus in on a specific type of data or date. Mesowx allows for multiple ways to access the data and even a way to input data into your MYSQL database, but I found that to be unrealiable and frankly risky to open your server to that function. We will give the Mesowx user just enough privilage to read data from the MYSQL database table and that is it. This limits the access that is available to be compramised from a web facing function.

Step 1: Requirments

Raspberry Pi with Weewx Weather Station running
MYSQL configured and operating
Download Mesowx to server

If you do not have mesowx on your server log in and download it:

wget https://bitbucket.org/lirpa/mesowx/downloads/mesowx-0.4.0.tar.gz

Untar it:

tar -xvzf mesowx-0.4.0.tar.gz

Step 2: Configure Server

Now copy the important parts to a web accessible directory:

cp -R mesowx-0.4.0/web /var/www/html/meso
cd /var/www/html/meso

We need to add a user to our MYSQL database to allow mesowx to access the data. Log into the MYSQL database as root:

mysql -u root -p

CREATE USER 'mesowx'@'127.0.0.1' IDENTIFIED BY 'password';
CREATE DATABASE weewx_raw;
GRANT SELECT ON weewx.* TO 'mesowx'@'127.0.0.1';
GRANT SELECT ON weewx_raw.* TO 'mesowx'@'127.0.0.1';
FLUSH PRIVILEGES;

There are two config files that must be edited to insert your MYSQL credentials.

cp meso/include/config-example.json meso/include/config.json 
nano meso/include/config.json

You need to modify the dataSource sections and make it match your MYSQL database information

"dataSource" : {
    "weewx_mysql" : { // the data source ID
        "type" : "mysql",
        "host" : "localhost",
        "user" : "mesowx",
        "password" : "password",
        "database" : "weewx"
},
    "weewx_mysql2" : { // the data source ID
        "type" : "mysql",
        "host" : "localhost",
        "user" : "mesowx",
        "password" : "password",
        "database" : "weewx_raw"

    }
}

Change the config file to match what you used when you modified your MYSQL database. You should also change the section where it checks to see if updates are allowed to the database.

"update" : {
                // set 'allow' to true to allow this entity's data to be updated
                "allow" : false,
                // the securityKey that must be sent along with each request to update
                // it's recommended to generate a random key, i.e. 
                // http://www.random.org/strings/?num=1&len=20&digits=on&upperalpha=on&loweralpha=on&unique=on&format=html&rnd=new
                // An empty key is not allowed.
                "securityKey" : "123"
            }

Save and close the file.

To finish up on the server you need to copy or rename the last file.

cp js/Config-example.js js/Config.js

Step 3: Configure RaspberryPi

You now need to log onto you weather station RaspberryPi to configure it to use "raw" data. This allows your website to have data more frequently then the normal five minute update. The update time will depend on the weather station, but mine is 50 seconds. Add THIS FILE to your $WEEWX_HOME/bin/user/. The file that is supplied by Mesowx WILL NOT work. I found this file after searching for a long time online in the google groups forums. I can not find where I got it from to give credit, but I didn't make it and it works!

First off we will stop weewx and add a database to MYSQL :

service weewx stop
mysql -u root -p


CREATE DATABASE weewx_raw;
GRANT select, update, create, delete, insert, drop ON weewx_raw.\* TO 'weewx'@'localhost';  
FLUSH PRIVILEGES;

Now we will make changes to the weewx configuration files. $WEEWX_HOME/weewx.conf*

Add this to the DataBindings :

[[raw_binding]]
    database = weewx_raw_mysql
    table_name = raw
    manager = weewx.manager.Manager
    schema = user.raw.schema

This to the Services section :

[[Services]]

    archive_services = weewx.engine.StdArchive, user.raw.RawService

Add this to the very bottom of your weewx.conf :

[Raw]

#
# This section is for configuration of the raw plugin. This plugin stores the raw
# data off of the station.
#
# The database binding to persist the raw data.
# This should match a section under the [DataBindings] section.
data_binding = raw_binding
#
# The max amount of raw data to retain specified in hours (set to None to retain all data)
# This will in effect keep a rolling window of the data removing old data based on
# the time of the most recent record. It is recommended to set this to at least 24.
#
# NOTE: if increasing this value (or use None to keep forever),
# keep in mind that raw data may consume VERY large amounts of space!
data_limit = 21

I would suggest restarting both weewx and MYSQL on the RaspberryPi and MYSQL on your server. My weewx.conf configuration file is available to review here

MYSQL Master Slave Replication over SSH tunnel

2017-04-22T16:00:00-05:00

The purpose of running having a master / slave relationship between your databases is that the slave database will be an exact replica of the specified master database. The purpose of running the connection through a SSH tunnel is to ensure:

Connection is encrypted
MYSQL server will only connect to the localhost
Minimal ports need to be opened at the firewall.

If you are unfamiliar with the SSH protocol or how to forward local ports, now is a great time to learn about it or to brush up on it from the Offical Website.

On Both Servers

On both the master and slave servers you must edit /etc/mysql/my.cnf

You must make the MASTERS'S server_id 1 and the SLAVEACCOUNT'S server_id = 2

server_id           = 1
log_bin             = /var/log/mysql/mysql-bin.log
log_bin_index       = /var/log/mysql/mysql-bin.log.index
relay_log           = /var/log/mysql/mysql-relay-bin
relay_log_index     = /var/log/mysql/mysql-relay-bin.index
expire_logs_days    = 10
max_binlog_size     = 100M
log_slave_updates   = 1
auto-increment-increment = 2
auto-increment-offset = 1

Restart MYSQL

service mysql stop
service mysql start

On Slave

adduser tunnelguy
usermod -s /usr/sbin/nologin tunnelguy
passwd -l tunnelguy

The purpose of changing tunnelguy's shell is to prevent the account from being used for anything else. With nologin set the account can not spawn an actual interactive shell, but still allows scp and forwarding local ports.

The last command passwd -l prevents tunnelguy from logging in with a password, but it is still allowed to authenticate via ssh keys. If you are unfamiliar with keyfiles then do a quick search online, there are a lot of tutorials showing how to create ssh keys. We will be need them in the next section.

On Master

mysql -u root -p
FLUSH TABLES WITH READ LOCK;
SHOW MASTER STATUS;
quit;
mysqldump -u root -p database > /tmp/weewx.sql
mysql -u root -p
UNLOCK TABLES;
GRANT REPLICATION SLAVE ON *.* TO 'slaveaccount'@'127.0.0.1' IDENTIFIED BY '<PASSWORD>';
quit;

There is a great package called autossh that creates a SSH tunnel that will reconnect on disconnect. This makes it great for unstable connections that will keep attempting to reconnect itself to the server.

apt-get install autossh

To make the ssh tunnel start on start up and when the network interface is up create a file in /etc/network/if-up.d/. In this case I specified the wlan0 interface.

vi /etc/network/if-up.d/autossh

/etc/network/if-up.d/autossh

#!/bin/bash
# Creates reverse tunnel through a remote server.
#USER=<Local Username>
#REMOTE_USER=<Remote Username>
#KEY=/home/you/.ssh/tun_rsa
#PORT=<Random High Port>
#HOST=<Address of Proxy Server>

# ssh options:
# -M: monitoring port
# -N: don't allocate a terminal
# -f: fork to background
# -q: quiet
# -R: reverse tunnel remoteport:host:localport
# -L: local port to forward
# -o: options
# -i: path to ssh key file

[ "$IFACE" = 'wlan0' ] || exit 0

su -c "autossh -M 0 -N -f -q -R 13306:127.0.0.1:3306 -L 13306:127.0.0.1:3306 -o ServerAliveInterval=60 -o ServerAliveCountMax=3 -o StrictHostKeyChecking=no -o BatchMode=yes -i /home/pi/.ssh/id_rsa tunnelguy@andrewhofmans.com"

Until you run that command in another terminal or restart the interface, it won't start an ssh tunnel.

Now that the Master is set up, we will move our database over to the slave.

scp /tmp/weewx.sql weewx@andrewhofmans.com:/tmp/weewx.sql

On Slave

Log into the MYSQL account and setup up the databases.

mysql -u root -p


CREATE DATABASE weewx;
GRANT REPLICATION SLAVE ON *.* TO 'slaveaccount'@'127.0.0.1' IDENTIFIED BY '<PASSWORD>';
quit;

Copy the Master database into the slave.

pv /tmp/weewx.sql | mysql -u root -p weewx

For *master_log_file='mysql-bin.000001', and master_log_pos=1234; ensure that you use the file and position from the Master from the beginning of the tutorial.

mysql -u root -p
CHANGE MASTER TO master_host='127.0.0.1', master_port=13306, master_user='slaveaccount', master_password='<PASSWORD>', master_log_file='mysql-bin.000001', master_log_pos=1234;
START SLAVE;
SHOW SLAVE STATUS\G
quit;

At this point the databases should be synced and everytime a entry is made into the master database it should be replicated into the slave database. Now that this is set up we will be able to set up a web application that utilizes our slave database to display our Weewx data.

Repair MYSQL Master Slave Replication

2017-01-21T02:29:00-06:00

One of the interesting parts of living on a boat is probably something that most people never think of dealing with. Power Management! Since we only have 30 amps of power available to draw at any one time we have to be quite careful on what is running. On those times that we aren't so careful and the breaker trips my Raspberry Pi running weewx loses power and the MYSQL connection to the web server is lost. This can lead to a corrupted database on the web server that will not work until the errors are cleared. One of the simplest ways that I found was to dump the MYSQL database on the Pi and then use that dump to correct the web server's slave copy. After doing this numerous times I decided there had to be a faster way of doing it instead of typing it out command by command. The following scripts are what I came up with and fix the database problems with running ONE command from the raspberry pi!

Change the -u(user) and "SuperSecretPass" to whatever your MYSQL credentials are.

I also used an identity file (passwordless login) for the scp command using a ssh key pair.

Script that runs on the Raspberry Pi

#!/bin/bash

#Stop weewx so we dont miss anything
service weewx stop

#Lock tables and set the current master log file and position into variables for use later
mysql -u root -p"SuperSecretPass" -e 'FLUSH TABLES WITH READ LOCK' weewx;
FILE=$(mysql -u root -p"SuperSecretPass" -e "SHOW MASTER STATUS;" -AN | awk '{print $1}')
POS=$(mysql -u root -p"SuperSecretPass" -e "SHOW MASTER STATUS;" -AN | awk '{print $2}')
echo $FILE > /tmp/file
echo $POS > /tmp/pos

#Dump the databases to file
mysqldump -u root -p"SuperSecretPass" weewx > /tmp/weewx.sql
mysqldump -u root -p"SuperSecretPass" weewx_raw > /tmp/weewx_raw.sql

#Unlock tables and restart the weewx service
mysql -u root -p"SuperSecretPass" -e 'UNLOCK TABLES' weewx;
service weewx start

#Copy current databases to remote server
scp -C -i /root/.ssh/www_RPi.key /tmp/weewx.sql weewx@andrewhofmans.com:/tmp/
scp -C -i /root/.ssh/www_RPi.key /tmp/weewx_raw.sql weewx@andrewhofmans.com:/tmp/
scp -C -i /root/.ssh/www_RPi.key /tmp/file weewx@andrewhofmans.com:/tmp/
scp -C -i /root/.ssh/www_RPi.key /tmp/pos weewx@andrewhofmans.com:/tmp/

#Execute the script on webserver
ssh -i /root/.ssh/www_RPi.key USER@andrewhofmans.com "sudo /home/USER/reset_database.sh"

#Clean up
rm /tmp/*.sql
rm /tmp/file
rm /tmp/pos

How to run sudo command without password

In the above script the "ssh -i /root/.ssh/www_RPi.key USER@andrewhofmans.com "sudo /home/USER/reset_database.sh" is able to be run as sudo without requiring the user to enter a password. To allow this action append

weewx   ALL=(ALL) NOPASSWD: /home/weewx/reset_database.sh

to your /etc/sudoers file.

Raspberry Pi runs this script on the webserver

#!/bin/bash

FILE=$(cat /tmp/file);  
POS=$(cat /tmp/pos);

mysql -uroot -p"SuperSecretPass" -e 'STOP SLAVE';  
pv /tmp/weewx.sql | mysql -uroot -p"SuperSecretPass" weewx  
pv /tmp/weewx_raw.sql | mysql -uroot -p"SuperSecretPass" weewx_raw  
mysql -uroot -p"SuperSecretPass" -e "CHANGE MASTER TO Master_Log_File='$FILE', master_log_pos=$POS;"  
mysql -uroot -p"SuperSecretPass" -e "START SLAVE;"  
mysql -uroot -p"SuperSecretPass" -e "SHOW SLAVE STATUS\G"  
rm /tmp/*.sql  
rm /tmp/file  
rm /tmp/pos

Lorex Technology ECO DVR

2016-05-30T18:48:00-05:00

While exploring my network I discovered that there was a securty vulnerability with my Lorex Technology ECO DVR. After doing more research and reaching out to the vendor and US-CERT I have finally documented and disclosed my first vulnerability report. It was a great learning experience working through the process and practicing responsible disclosure. Unfortunately with cve.mitre.org changing its coverage goals it did not get a CVE identifier. I submitted my report to Bugtraq and the original can be found here or here.

ADVISORY INFORMATION

Product: Lorex Technology ECO DVR
Vendor URL: https://www.lorextechnology.com/
Type: Hard coded password [CWE-259]
Date found: 2016-05-04
Date published: 2016-05-30
CVE: -

VERSIONS AFFECTED

I have successfully tested the vulnerability on a Lorex Technology LH162400 DVR with the most current firmware (V5.2.0-20141008). I suspect that all DVRs that can utilize Lorex Stratus Client and Lorex ECO Stratus Android app will be vulnerable. Affected DVRs likely include the vendors and versions listed specifically in the code.

INTRODUCTION

LOREX provides businesses and consumers with professional-grade DIY video surveillance systems and plug and play wireless video monitoring solutions.

(from the vendor's homepage)

VULNERABILITY DETAILS

Remote access to the device is possible using Lorex's Stratus Client which is downloadable from the vendor. User is prompted for IP, username/password, and port. DVRs are easily identified on a LAN using normal port scanning and enumeration. Default username and password is admin:000000 (from manufacturer manual). On first login admin user is prompted to change password. No matter what the password is or what it is changed to the "SuperPassword" grants admin access to the device.

The following Proof-of-Concept is found in plaintext in the \installation directory\new-trunk\js\main.js :

function CheckPassword(){};  
$(function(){  
$("btn_reboot_ok").click(function(){  
var SuperPassword;  
if(gDvr.nMainType == 0x52530003 || (gDvr.nMainType == 0x52530002 &&
    gDvr.nSubType == 0x50100) || (gDvr.nMainType == 0x52530000 &&
    gDvr.nSubType == 0x60300)){  
    SuperPassword = "130901";  
    }else{  
    SuperPassword = "070901";  
}  
if(lgCls.version == "SWANN"){  
    SuperPassword = "479266";  
    }
    else if(lgCls.version == "PROTECTRON"){  
    SuperPassword = "Ab9842";  
    }  
    if($("reboot_input").val() == gVar.passwd ||
    $("reboot_input").val() == SuperPassword){  
    MasklayerHide();  
    $("reboot_prompt").css("display","none");  
    CheckPassword();  
}

RISK

To successfully exploit this vulnerability an attacker must have remote access to the DVR over port 9000. Attacker can use Lorex's Stratus Client and use the hardcoded admin password for specific vendor and model.

The vulnerability allows remote attackers full administrative access to the device.

SOLUTION

To truly fix the vulnerability a fix must come from the vendor. Updated firmware is not available and when questioned the vendor does not respond. Best fix for now is to mitigate the problem. Prevent remote access to port 9000 at the firewall. Segregate DVR from normal LAN to limited access internal LAN segment / VLAN.

REPORT TIMELINE

2016-05-04: Discovery of the vulnerability
2016-05-05: Informed applicable Vendors
2016-05-05: Submitted vulnerability to US-CERT
2016-05-05: Response from US-CERT informing similar vulnerability was previously reported which vendor ignored. No further attempts will be made.
2016-05-16: Response from Swann
2016-05-30: Advisory released

Raspberry Pi WS-2080 Weather Station

2016-04-25T11:19:00-05:00

After surviving a windstorm that produced 40-60 mph winds on the boat I decided that I wanted a weather station to capture that data! The closest anemometer was very far away and definitely was not subjected to the same winds that I had in my marina. The following post is the steps I took to get my system up and running. It will probably be a work in progress because I like to tinker and fix things until they are broken.

For my overall design I had to make a decision right off the bat on whether I wanted to build the system hardware from the ground up or to use a complete commercial unit and focus on getting the code to work. After a lot of research I decided to go with the commercial unit. This is mainly to do with my current living situation and not having an easy way to get power and sensor readings from a remote sensor unit. In the future I would like to build the sensor array myself and get everything running off of solar, but that will be a ways off.

Working Weather Station

Real Time Update and Graphing

Step 1: Gather parts

Ambient Weather WS-2080

This weather station is not being sold for the $86 that I got it for and may not be the best option anymore. The Ambient Weather WS-2095 is supposed to be same / similar and may be better. It does have the sensors integrated together and not wired separately so that will be an issue if you decide to connect them directly to a custom pcb or a Raspberry Pi instead of using the base station.

Raspberry Pi 2

For the Raspberry Pi I decided to go with the Raspberry Pi 2 Model B. It has plenty of power and has the required number of USB ports to interface with WiFi dongle and the weather base station. The Rpi 3 is definitely an option now since it has built in WiFi capabilities. The link above is a kit, you can individually source the components, prices may vary.

8+ GB Micro SD Card (Class 10)

I used the card that came in the kit, but I will probably upgrade this soon. I made it work as described below, but it would have saved me some time and frustration if I had bought a larger SD Card right away.

Step 1.5 : Download Software

Depending on size of SD Card and success setting everything up you may want re-flash and just put Rasbian on the card right away.

Rasbian Jessie Lite
SDFormatter
Win32DiskImager

Step 2: Some Assembly Required

Once everything arrived it was really easy to get it together. The weather station went together simply by bolting and screwing the sensors to the mounting bracket. I ended up putting the assembled sensor array on my fence temporarily while I got everything working together. For the Pi, it fit right into the case and all the ports lined up. I inserted the 8 GB SD Card from the kit that was already loaded with NOOBS and booted it up. In the middle of updating and downloading the required software to interface with the weather station I ran out of room on the SD Card! This really surprised me considering I had barely started and I didn't have a database full of data. I decided to get and install Rasbian Jessie Lite. After downloading I figured how to clear and then flash the SD Card using SDFormatter and Win32DiskImager. I had to use the SDFormatter because Windows 7 was not seeing the SD Card as 8 GB. It only said that there was 400 MB remaining on the card. Once it was reformatted I used Win32DiskImager to reflash the card with my new Rasbian Lite image. These tools were pretty self explanatory and had an easy GUI to navigate.

Step 3: Update and Install

So now I have the SD Card freshly flashed with Rasbian Lite and installed in the Pi. I connect to it and log in using the default Username: pi Password: raspberry. The following should get the Pi up to date and ready to install the remaining software.

apt-get update  
apt-get upgrade  
apt-get dist-upgrade

This is directly from the Weewx site and are required dependencies:

apt-get install python-configobj  
apt-get install python-cheetah  
apt-get install python-imaging

These are also required if hardware is serial or USB:

apt-get install python-usb

If you are going to be using MySQL also install the following: Remember the root MySQL Password when you set it.

apt-get install mysql-client mysql-server  
apt-get install python-mysqldb

Also required if you are on Raspbian and planning on using FTP:

apt-get install ftp

These are optional for extended almanac information:

apt-get install python-dev  
apt-get install python-pip  
pip install pyephem

We need to add a Weewx user. You can take all the defaults when prompted:

adduser weewx

Create the MySQL Database that we are going to use:

mysql -u root -p  
CREATE DATABASE weewx;  
use weewx;  
CREATE USER 'weewx'@'localhost' IDENTIFIED BY 'putyourpasswordhere';  
GRANT select, update, create, delete, insert, drop ON weewx.\* TO weewx@localhost;  
FLUSH PRIVILEGES;  
quit;

Now go to Weewx and find out what the newest version is. Once you find the url to the newest version:

cd /home/weewx  
wget http://weewx.com/downloads/weewx-3.5.0.tar.gz

Unzip and install:

tar xvfz weewx-X.Y.Z.tar.gz  
cd weewx-3.5.0  
./setup.py build  
./setup.py install

During the actual install it will ask for some information about the station. Google maps is helpful to find your lat and long.
Before we start the program we need to edit the weewx.conf file.

nano weewx.conf

It is important to change line 10 so that it reads "database = archive_mysql" Find this section and modify it as needed to match your MySQL install:

##############################################################################

#   This section binds a data store to a database.

[DataBindings]

[[wx_binding]]
    # The database must match one of the sections in [Databases].
    # This is likely to be the only option you would want to change.
    database = archive_mysql
    # The name of the table within the database
    table_name = archive
    # The manager handles aggregation of data for historical summaries
    manager = weewx.wxmanager.WXDaySummaryManager
    # The schema defines the structure of the database.
    # It is *only* used when the database is created.
    schema = schemas.wview.schema

##############################################################################


##############################################################################

# This section defines various databases.

[Databases]

# A SQLite database is simply a single file  
[\[archive\_sqlite\]\]  
database\_name = weewx.sdb  
database\_type = SQLite

# MySQL  
[\[archive\_mysql\]\]  
database\_name = weewx  
database\_type = MySQL



##############################################################################

#This section defines defaults for the different types of databases.

[DatabaseTypes]

# Defaults for SQLite databases  
[[SQLite]]  
driver = weedb.sqlite  
# Directory in which the database files are located  SQLITE\_ROOT = %(WEEWX\_ROOT)s/archive

# Defaults for MySQL databases  
[[MySQL]]  
driver = weedb.mysql  
# The host where the database is located  
host = localhost  
# The user name for logging in to the host  
user = weewx  
# The password for the user name  
password = weewx  
##############################################################################

We are going to want this to run everytime the Pi turns on so lets daemonize this:

cd /home/weewx  
cp util/init.d/weewx.debian /etc/init.d/weewx  
chmod +x /etc/init.d/weewx  
update-rc.d weewx defaults 98

Prior to running it for the first time, I modified the weewx.conf and changed "debug = 0" to "debug = 1" so that I would have more verbose output and better error messages. Before we run it for the first time, it is a good idea to open up another terminal to the Pi and watch the log file as it starts. So in a new terminal on the Pi run:

sudo tail -f /var/log/syslog

In the first terminal run:

service weewx start

If all goes well it should be running with no errors. In roughly 5 minutes it will create a report and put it in /home/weewx/public_html

You view or move these reports from there. If you just want local access to them you can install a webserver on the Pi.

apt-get update  
apt-get install nginx  
cd /var/www/html
ln -s /home/weewx/public_html weather

In a web browser navigate to http://yourraspberrypiIPaddress/weather

Working default example

A much nicer and dynamic display: Dynamic display uses Mesowx . I will be doing a follow on post to explain the process I used to get that going.

The following resources were extremely helpful in getting this project working and creating this guide.
WEEWX

C++

2016-02-19T06:22:00-06:00

These posts will be various lessons and lessons learned as I start learning how to program in C++. My first experiences programming started in Windows batch files and then in high school I was introduced to Perl. While it could be argued that neither batch or Perl are "real programming languages" they non-the-less introduced me to the world of programming.

Useful C++ Resources

The following links are the sites that I found most helpful for me. I have found that I learn best by reading the text, looking at examples of code, and then systematically analyzing the code and breaking it down into smaller parts. I study the individual parts until they make sense and I UNDERSTAND what / how the statement works, not just that it does.

cplusplus.com

cprogramming.com

Boat

2015-11-09T04:11:00-06:00

While living in Washington, my wife and I decided to buy a boat and live on the water. Here is our 1985 President 41 that we and our cat lived on for 3 years in Kenmore, WA.

Useful Boat Project Resources

I have some of my favorite resources listed below. Some were better then others and some just gave me some great ideas of what is possible.

docksidereports.com
yachtsurvey.com
boatus.com
trawler-beach-house.blogspot.com