IPv6 Multicast Listener for Elixir

I have yet to find a working example of this, so I am obligated to post one. Especially considering the frustration I and a colleague more intelligent than I had to undergo before we found a working solution.

Please note that you will not get away with using an old version of OTP. IPv6 Multicast joins were broken for a long time.

Here is a simple script that will listen for a single packet, print it out, and exit.

require Logger
address={0xff1e, 0, 0, 0, 0, 0, 0x70, 0x25}
iface=0
{:ok, socket} = :gen_udp.open(8208, [
{:ip, address},
{:add_membership, {address, iface}},
{:debug, true},
:binary,
{:reuseaddr, true},
{:recbuf, 8388608},
])
Logger.info("UDP socket opened successfully #{inspect(socket)}")
{:ok, message} = :gen_udp.recv(socket, 9)
Logger.info("Received message: #{inspect(message)}")

An interface index of 0 works for global multicast addresses. Your kernel should figure it out based on the routing table. For a link-local or unique-local address, you will need to set that to your actual interface index.

NOTE THE ORDER OF THE OPTIONS

If you move :add_membership lower down, you will get a nonsensical eaddrinuse error. Yeah, that took a while to figure out. The other options can be moved around, but add_membership needs to be up towards the top. We assume this is because the options are processed in the order in which they are provided instead of the order of which they should be.

So let's move on to the gen_server implementation to expose the next gotcha.

defmodule Listen do
use GenServer
require Logger
@address {0xff1e, 0, 0, 0, 0, 0, 0x70, 0x25}
@iface 0
#@address {224,0,0,1}
#@iface {0,0,0,0}

def start_link(_) do
GenServer.start_link(__MODULE__, {}, name: __MODULE__)
end

def init(_opts) do
{:ok, socket} = :gen_udp.open(8208, [
{:ip, @address},
{:add_membership, {@address, @iface}},
{:debug, true},
:binary,
{:reuseaddr, true},
{:recbuf, 8388608},
{:active, true}
])
Logger.info("UDP socket opened successfully #{inspect(socket)}")
{:ok, %{socket: socket}}
end
def handle_info(msg, state) do
Logger.info("Received message: #{inspect(msg)} #{inspect(state)}")
{:noreply, state}
end
end
Listen.start_link([])
Process.sleep(:infinity)

Here, you must add {:active, true}, or handle_info() will never be called. Elixir will just leave the packets in the network queue. This is not true for IPv4. I left in the commented out IPv4 options if you want to prove this to yourself.

I would be remiss if I didn't provide you a command to test these.

echo "Elixir sucks" > /dev/udp/ff1e::70:25/8208

Before I leave, I will urge you to hold off on using elixir for important things. It is an immature pet project with poor performance, and you will be pressed to find engineers well versed in it who can maintain it with any efficiency.

Remote Log File Monitoring with Bash and Netcat

Yeah, you can probably just shell right up in there.

ssh root@obscure-server.com tail -f /var/log/syslog

But sometimes you just need those logs to come straight to you. Maybe to jump onto a different network.

We still have to shell in briefly to kick it off of course, but that only takes a moment.

I present to you, a script named fu (file over udp).

#!/bin/bash
LAST=$(tail -n 1 $1)
while [ 1 ]
do
sleep 0.0001
CUR=$(tail -n 1 $1)
if [ "$LAST" != "$CUR" ]; then
LAST=$CUR
printf '%s\n' "$LAST" > "/dev/udp/$2/$3"
fi
done

Dumb, insecure, and full of ignored edge cases, but it works just well enough to not warrant another ounce of effort.

We can shell in, spawn it, and leave:

nohup ./fu /var/log/syslog obscure-client.com 12345 &

And presto, obscure-server.com will start forwarding new lines appended to /var/log/syslog to port 12345 of  obscure-client.com until death.

Then on obscure-client.com, we can use netcat to print out the new log lines as they as added. 

nc -lukvw 0 12345

Of course, it's way more useful to simply spawn a persistent reverse SSH tunnel instead of fu instance.

ssh-copy-id root@obscure-client.com
nohup ssh -N -R 8000:localhost:22 root@obscure-client.com &

Then you can ssh from obscure-client.com to obscure-server.com whenever you want, regardless of those pesky firewalls.

ssh localhost -p 8000 
Seems like stuff every hacker knows by heart, but I am no hacker.

Prevent FFMPEG from removing KLV data stream from MPEG-TS output

ffmpeg is a great tool for recording live video streams to files, clipping video files, streaming video files to udp, and a bunch of stuff I don't care about. Of course, I feel like it's always a battle to get the right flags to do exactly what I need.

The most recent example, I just want to copy an MPEG-TS file:

ffmpeg -i ./CheyenneVAhospital.ts out.ts

Yeah, of course it isn't that easy. ffmpeg has decided that I have no need for the KLV data stream.

Input #0, mpegts, from './CheyenneVAhospital.ts':

  Duration: 00:01:05.15, start: 271.125322, bitrate: 5535 kb/s

  Program 1 

  Stream #0:0[0x100]: Video: h264 (Baseline) ([27][0][0][0] / 0x001B), yuv420p(progressive), 1920x1080 [SAR 1:1 DAR 16:9], 29.97 fps, 29.97 tbr, 90k tbn, 59.94 tbc

  Stream #0:1[0x101]: Audio: aac (LC) ([15][0][0][0] / 0x000F), 48000 Hz, stereo, fltp, 127 kb/s

  Stream #0:2[0x102]: Data: klv (KLVA / 0x41564C4B)

Stream mapping:

  Stream #0:0 -> #0:0 (h264 (native) -> mpeg2video (native))

  Stream #0:1 -> #0:1 (aac (native) -> mp2 (native))

Press [q] to stop, [?] for help

Spoiler: I do need it. Very much. And after trying many flags (codec:data, map_metadata, copy_unknown, etc.), I found that I had to explicitly map all three stream types.

ffmpeg -i ./CheyenneVAhospital.ts -map 0:v -map 0:a -map 0:d out.ts

Input #0, mpegts, from './CheyenneVAhospital.ts':
  Duration: 00:01:05.15, start: 271.125322, bitrate: 5535 kb/s
  Program 1 
  Stream #0:0[0x100]: Video: h264 (Baseline) ([27][0][0][0] / 0x001B), yuv420p(progressive), 1920x1080 [SAR 1:1 DAR 16:9], 29.97 fps, 29.97 tbr, 90k tbn, 59.94 tbc
  Stream #0:1[0x101]: Audio: aac (LC) ([15][0][0][0] / 0x000F), 48000 Hz, stereo, fltp, 127 kb/s
  Stream #0:2[0x102]: Data: klv (KLVA / 0x41564C4B)
File 'f.ts' already exists. Overwrite? [y/N] y
Stream mapping:
  Stream #0:0 -> #0:0 (h264 (native) -> mpeg2video (native))
  Stream #0:1 -> #0:1 (aac (native) -> mp2 (native))
  Stream #0:2 -> #0:2 (copy)
Press [q] to stop, [?] for help

Simple enough, but I am compelled by future me to preserve this knowledge in the cloud so that I can further delay defragmenting my brain.

Using multicat to send an MPEG-TS video file to a UDP multicast address

Multicat is fairly simple to use. Just give it an input and output and it figures the rest out on it's own. 

multicat -uU ./CheyenneVAhospital.ts 226.6.6.1:8209

The -uU flags to ignore the missing RTP headers.

But wait! You need a .aux file apparently

error: couldn't open file ./CheyenneVAhospital.aux (No such file or directory)

Piece of cake. The docs say you just have to ingest the file with the ingests command. Let's copy and paste that into our shell

ingests -p 68 ./CheyenneVAhospital.ts

Oh great. That didn't work.

debug: end of file reached

error: no PCR found

Alright our PCR PID must be wrong. Luckily we can use ffprobe to get the PCR PID out of a .ts file.

ffprobe CheyenneVAhospital.ts -show_programs | grep pcr_pid

...

pcr_pid=256

Oh lovely! Why can't ingests do that? I dunno man stop asking questions.

Let's try ingests with the correct PCR PID

ingests -p 256 ./CheyenneVAhospital.ts

Hey, cool. No errors this time. And our .aux file is more than 0 bytes!

Now we can finally do the thing.

multicat -uU ./CheyenneVAhospital.ts 226.6.6.1:8209

Oh by the way, if you want to change your MPEG-TS file to have a consistent video bitrate, you can use ffmpeg for that. Here's an example of converting the video stream to 1Mb/s

ffmpeg -i ./CheyenneVAhospital.ts -c:v libx264 -b:v 1M -map 0 CheyenneVAhospital_1M.ts

And multicat doesn't support looping, so you'll just have to put the multicat command inside a loop body and deal with the spikes

for (( ; ; ))

do

multicat -uU ./CheyenneVAhospital_1M.ts 226.6.6.1:8209

done

VBoxManage: error: The virtual machine 'minikube' has terminated unexpectedly

Full error: 

Exiting due to PR_VBOX_BLOCKED: Failed to start host: creating host: create: creating: Unable to start the VM: /usr/bin/VBoxManage startvm minikube --type headless failed:

VBoxManage: error: The virtual machine 'minikube' has terminated unexpectedly during startup with exit code 1 (0x1)

VBoxManage: error: Details: code NS_ERROR_FAILURE (0x80004005), component MachineWrap, interface IMachine


Solution:

# Purge VirtualBox.

sudo killall VirtualBox

sudo apt-get remove --purge virtualbox

rm -rf ~/.config/VirtualBox/


# Wipe minikube cache

rm -rf ~/.minikube/


# Reinstall VirtualBox

sudo apt-get install virtualbox


# Rebuild minikube VM

minikube start --driver=virtualbox


Solution in one line:

sudo killall VirtualBox; rm -rf ~/.config/VirtualBox/; rm -rf ~/.minikube/; sudo apt-get remove -y --purge virtualbox && sudo apt-get install  -y virtualbox && minikube start --driver=virtualbox



Convert PostGIS ST_LineString to ST_MultiPoint

Sometimes you need to convert a PostGIS line into a multipoint geometry. I mean, how else did you find this?

SELECT ST_Union(

    ARRAY(

        SELECT (ST_DumpPoints(linestringcolumn)).geom

        FROM atable

    )

);

Just kidding, use ST_Points(). Why ST_MultiPoint() does not do this is a mystery to my tiny brain. I would ask Dan Baston, but his answer would probably fracture my mental model of reality, and I don't have time to reconstruct it right now.

SELECT ST_Points(linestringcolumn) FROM atable;


Fix for npm ERR! command-line line 0: "unsupported option accept-new"

I encountered this on a server with a newer npm version and an older openSSH version. A few node dependencies were stored in private git repositories, and npm just simply wouldn't fetch them.

The fix was to downgrade npm to v6

    npm i -g npm@6


Unreal Engine 4 pointer cursor stuck on web browser widget

I know UE4's web browser widget is an "experimental" feature, but this is a pretty obnoxious bug that they have refused to fix for at least 4 years.

As far as I can tell, as soon as your cursor enters the "pointer" state while inside the browser widget, it will be stuck there indefinitely, even after you close the widget. The only solution I have found is to prevent it from ever reaching that state.

If you have full control over the webpage you are displaying, you can modify your CSS to use some other cursor type for all your elements. I did not.

My solution? Set the cursor for every element to 'default' using javascript.
var nodes = document.querySelectorAll('*');
for(var i=0; i<nodes.length; i++) {
    nodes[i].setAttribute('style', 'cursor:default !important'); 
}

When? All of the time.

It does seem excessive to do every element every tick, and it probably is, but when you have no control over a webpage, you never know when some new pesky element is going to be shoved into the DOM and screw you again. Why risk it?

One thing I did try was to set a software "Hand" cursor. I assume this translates to cursor:pointer, but who knows? That did nothing at all. Maybe the bug here is that there is no "pointer" cursor type in UE4, so the web browser widget has no idea how to turn it off.

One thing I did NOT try was to set up invisible hardware cursor images. Perhaps there is a conflict in the transition from hardware to software cursors? I do not know, but I do know that I end up with both when I mouse out of an HTML button. Forever.


Reducing Wordpress, PHP, MySQL Memory Usage

BACKGROUND


I hate WordPress, but I also love WordPress. It is still the most user-friendly open source blogging platform in existence. I have tried many alternatives, I've even written my own, but the feature set has always felt lacking.

My main problem with WordPress used to be the codebase. It was a mess, but the devs have managed to clean it up quite a bit. The current version seems quite tidy as far as PHP projects go, and hacking on it isn't nearly as frustrating as it had been in earlier versions.

So now my big beef is with performance. PHP and MySQL are slow, gobble up tons of resources even when sitting idle, and WordPress' extensive feature set dumps a ton of bloat on top.

At the end of the day, I just want users to load my dumb blog as fast as possible and have WordPress leave as small of a footprint as possible so I can save my server resources for more interesting projects.


HOW TO REDUCE MYSQL MEMORY USAGE


The first thing I notice when checking my system memory is MySQL. Why are you eating half of my memory, MySQL? You're not doing anything. Just give it back.

MySQL loves to cache. It increases query speed tremendously to store your entire database in memory, but that's not my goal. Once I'm done writing my post, I won't need MySQL anymore, because I will be generating a static version of my blog for public viewing. Nothing serves faster than static files, and I intend my blog to be read only for the general public.

My end goal now is to dial down MySQL's memory usage as far as possible while still being able to use the Wordpress admin panel without noticeable lag.

So open up your MySQL config (mine was at /etc/mysql/mysql.conf.d/mysqld.cnf) and start tweaking some settings.

The biggest single improvement that I was able to make was to turn off MySQL's Performance Schema by adding this line:

performance_schema = 0

I also found a huge list of variables here and then proceeded to completely ignore the "minimum" values:

key_buffer_size = 8
max_allowed_packet = 1K
max_connections = 10
query_cache_limit = 0
query_cache_size = 0
expire_logs_days = 1
max_binlog_size = 1K
thread_stack = 1
thread_cache_size = 0
host_cache_size = 0
innodb_buffer_pool_size = 0
innodb_log_buffer_size = 0
innodb_ft_cache_size = 0
innodb_ft_total_cache_size = 0
sort_buffer_size = 0
read_buffer_size = 0
read_rnd_buffer_size = 0
max_heap_table_size = 0
tmp_table_size = 0
bulk_insert_buffer_size = 0
join_buffer_size = 0
net_buffer_length = 0
innodb_sort_buffer_size = 0
binlog_cache_size = 0
binlog_stmt_cache_size = 0

Now just save and restart MySQL.

I'm amazed that WordPress can still function at all with these settings, but honestly I don't even notice the difference when clicking around the admin panel. But what I did notice what that I had slashed MySQL's memory usage by 90%.


HOW TO REDUCE PHP MEMORY USAGE


Just like MySQL, I will only need PHP to write posts, not to serve them. Again, I will be minimizing PHP's memory usage by modifying configuration files.

PHP's biggest issue is that, by default, it spawns multiple processes to handle incoming requests, and then refuses to kill them once the requests have been handled. These processes, even when sitting idle, take up way too much of my precious memory.

So most of the work can be done in the pool config. For my system, that file is at /etc/php/7.0/fpm/pool.d/www.conf

pm = ondemand
pm.process_idle_timeout = 1s

This configuration tells PHP to not create any listener processes until a request is actually made. If a process does nothing for more than 1 second, it will be killed. Humanely. (PETA plz no bully.)

So PHP can now eat all the memory it wants when I'm actually using it (almost never) and will give it back immediately after. If you are worried about peak memory usage, you can set 

pm.max_children = 1

Other than than you can't do much. Limiting a PHP thread's max memory size will result in errors. You are essentially at the mercy of WordPress' dev team.


HOW TO REDUCE WORDPRESS MEMORY USAGE


All of this is has been a lead up to the main objective, building WordPress into a static website. Static websites are faster, more stable, and more secure. They also require very little memory to serve.

If you don't mind manually generating your public-facing WordPress site, I would recommend installing the Simply Static WordPress Plugin, and just use that.

If you're lazy, I'll share how I was able to trigger a static site generation every time a post is published or updated.

The first step is to create a shell script to clone your WordPress site into a public directory. I called mine static.sh

# Download your dynamic wordpress site into /tmp
wget --trust-server-names --no-clobber --mirror --convert-links --adjust-extension -e robots=off -P /tmp http://privateblog.com

# Search for any mention of your private domain and replace it with your public domain
find /tmp/privateblog.com -type f -print0 | xargs -0 sed -i 's/privateblog\.com/publicblog\.com/g'

# Manually copy your uploads directory to make sure you have all the images sizes that wget ignored
cp -r /var/www/privateblog.com/wp-content/uploads/* /tmp/privateblog.com/wp-content/uploads/

# Move the result into your public www directory

rm -rf /var/www/publicblog/*
mv /tmp/privateblog.com/* /var/www/publicblog/

Now we have a bash script that needs to run whenever we update our blog. Luckily WordPress makes this easy by shipping with XML-RPC functionality.

Since XML-RPC uses HTTP as a transport protocol, we don't need any extra technology to detect a change; we can just use PHP. Of course, if you wanted to analyze the XML payload, you would need to set up an actual XML-RPC server, but that's not necessary here.

So now we just need a PHP script added to our WordPress directory which calls static.sh. I called mine rpc.php

<?php passthru('./static.sh'); ?>


Nice. The last piece of this is to add the URL of rpc.php to our WordPress Update Services. In the dashboard, go to Settings > Writing and find Update Services down at the bottom. Add your URL to the box, and hit Save Change.


That should do the trick. If you run into problems, double check your file permissions and ownership. Keep in mind that the www-data user needs to be able to execute both scripts and have write access to your public directory.

Disable $mdBottomSheet Drag Events

I recently ran into a situation where I needed to prevent an md-bottom-sheet component from dragging around. Inside the bottom sheet, we have an image cropper component (ui-cropper) which has it's own drag events. When we drag the cropper window up, the entire bottom sheet moves up with it. When we drag the cropper window down, the bottom sheet closes.


Obviously a bottom sheet isn't the best element to contain an image cropper, but hey, I try to avoid arguments with my UX team when possible.

Now because the version of Angular Material we are using doesn't support drag disabling for md-bottom-sheet, we have to hack around it. I'm okay with a little bit of hacking, but I'm certainly not going to modify Angular Material's core files over a questionable UX decision.

My compromise is to add a bit of code to our directive which intercepts and neutralizes $mdDrag events. Five lines of code for a one-off conflict doesn't seem like the end of the world.

The directive I'm modifying is nothing more than a button which opens a bottom sheet with a small form in it. All I need to do is attach a short function disableDrag to the scope which would is triggered when the bottom sheet is clicked.

function link(scope) {

  scope.disableDrag = disableDrag;
  scope.onClick = onClick;
    
  function disableDrag() {
    $('md-bottom-sheet') // find the bottom element
      .on('$md.drag', e => false) // hold it still
      .on('$md.dragend', e => false); // prevent closing
  }
    
  function onClick() {
    $mdBottomSheet.show({ 
      templateUrl: require('./bottomSheet.html') 
    }).then(() => {
      // process form data
    });
  }
}

Now all I had to do was link it to the bottom sheet element.

<md-bottom-sheet ng-mousedown='disableDrag()'>
  <!-- our form -->
</md-bottom-sheet>

Works like a charm.

Stop SVGO and GULP-SVGMIN from Breaking Paths and Colors in SVG Images

When practical, I like to use vector images on my websites. They are great for responsive design, and look much more clean and crisp than PNG or JPG images. Of course, I am also a fan of performance, and SVG files can often come significantly larger than they need to be.

For JPG and PNG images, I use a tool called Trimage to compress them. For SVG files, I use SVGO. Actually, I use the gulp package gulp-svgmin, but that is just a gulp wrapper for SVGO.

The problem with SVGO, is that with the default settings, it tends to break many SVG images that come out of tools like Adobe Illustrator. For your benefit, I have trial-and-error'd my way through all of the SVGO plugins and isolated the options which cause issues.

If your SVG image paths are breaking, turn off convertPathData and mergePaths.

If your colors are turning black or being removed, turn off removeUnknownsAndDefaults.

There have also been reports of issues with collapseGroups, but I have not had an issue yet.

Here is a snippet from the gulp file I am currently using:
var gulp = require('gulp');
var svgmin = require('gulp-svgmin');
var debug = require('gulp-debug');

gulp.task('compress', function () {
  return gulp.src('./public/img/**/*.svg')
    .pipe(debug())
    .pipe(svgmin({plugins: [
      { convertPathData: false } // breaks paths
      , { mergePaths: false } // breaks paths
      , { removeUnknownsAndDefaults: false } // breaks colors
    ]}))
    .pipe(gulp.dest('./public/img/'));
});

SVG image files may still be larger than PNG counterparts, but I find that using a gzip compression tool like the nodejs compression package can often crunch these optimized SVG files another 300%, making them significantly smaller than gzipped PNG files, and they look so much better!

Securing MongoDB with Authentication and Authorization using Node.JS and Mongoose on Ubuntu

One of the nice things about MongoDB is that it doesn't force your to deal with account management; you simply install it and you can immediately start cramming it full of data anonymously.

But if you need to access a database remotely, or you just want an extra layer of security for your data, setting up MongoDB user accounts and restricting access is very simple and highly recommended.

Step 1) Create an administrator account

It's a swell idea to create an account with access to all databases, and then use this account to create more restricted accounts for users and applications.

Each user is stored in and authenticated against a single database, though you can grant it access to other databases using the 'roles' attribute. When creating a user, it will be created in the current database, so let's switch to the 'admin' database to store our admin user in:

use admin;

To create a user in the currently selected database, we call db.createUser() and specify 'user', 'pwd', and 'roles' attributes. To create a root user (one with full privileges across all databases) named 'admin' with password 'pass':

db.createUser({
  user : "admin",
  pwd  : "pass",
  roles: [
    { 
      role: 'root', db: 'admin'
    }
  ]
});

You can see that each role object in the roles array requires a role name and a database name. For some reason, you still have to specify a database even if the role allows access to all databases.

There are a ton of Built-in MongoDB user roles, and they will meet most of your needs. If they do not, please be aware that you can create your own custom user roles. Useful roles include 'read', 'readWrite', 'dbAdmin', 'userAdmin', 'backup', 'restore', and 'root'

Step 2) Test your administrator account

mongo admin -u admin -p pass

Even though this user has full access to all databases, authentication will fail if we do not connect to the admin database. This is because the admin user is stored in the admin database, and mongo needs to be told which database the user is stored in before it can authenticate them. By default, mongo assumes the user is stored in the database that you are connecting to, but you can override this assumption by using the --authenticationDatabase flag

mongo -u admin -p pass --authenticationDatabase admin

Step 3) Create a restricted user for your application

It's not the most wise strategy to use your root account for your application. Each user and application should have their own account that is restricted to access only the data that they need to function.

For this we will assume that you have a database named 'app' that you use for your web application, and your application needs to read and write to 'app'.

use app;

db.createUser({
  user : "appuser",
  pwd  : "apppass",
  roles: [
    { 
      role: 'readWrite', db: 'app'
    }
  ]
});

Step 4) Force authorization

Right now, your mongo database accepts authentication, but authentication is not required. In order to force users to authenticate, we have to enable it in the /etc/mongod.conf configuration file

FOR MONGODB 3.0 AND ABOVE add these lines:

security:
  authorization: enabled

FOR MONGODB 2.X add this line:

auth = true

Any changes to your config file require a restart before they will take effect:

sudo service mongod restart

Step 5) Connect with Node.js / Mongoose.js

var fs = require('fs')
  , mongoose = require('mongoose')
  , mongoUri = "mongodb://appuser:apppass@127.0.0.1:27017/app"
  ;

mongoose.connect(mongoUri);

All of the magic here is contained in the mongo connection string. This same connection string also works with the native mongodb driver.

Here we are connecting the same database that the user is stored in (app), so it should authenticate properly. To connect to a different database that the user is not stored in, we can add the 'authSource' attribute. Here we are connecting to the 'app2' database with 'appuser' who is stored in the 'app' database:

mongodb://appuser:apppass@127.0.0.1:27017/app2?authSource=app"

Step 6) [Optionally] allow external connections

If you need to connect to your database from another machine in the network, you have to tell mongo to bind to an external IP address.

FOR MONGODB 3.0 AND ABOVE comment the bindIp line:

net:
  # bindIp: 127.0.0.1

FOR MONGODB 2.X comment the bind_ip line:

# bind_ip = 127.0.0.1

And restart:

sudo service mongod restart

Step 7) [Optionally] encrypt your MongoDB traffic with TLS/SSL

Connecting to MongoDB over TLS/SSL with Node.JS and Mongoose on Ubuntu

Step 1: Obtain MongoDB 3.0

The first thing you need to know is that that SSL is only supported out-of-the-box by MongoDB 3.0 and later. Ubuntu does not have 3.0 in the default repositories, so here's how you get it:

sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 7F0CEB10
echo "deb http://repo.mongodb.org/apt/ubuntu trusty/mongodb-org/3.0 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-3.0.list
sudo apt-get update
sudo apt-get install -y mongodb-org=3.0.7 mongodb-org-server=3.0.7 mongodb-org-shell=3.0.7 mongodb-org-mongos=3.0.7 mongodb-org-tools=3.0.7

3.0.7 is the latest stable version as of now, but feel free to substitute 3.0.7 with your favorite release.

Step 2: Obtain Private Key, Certificate, and PEM files

The PEM contains a Public Key Certificate and its associated Private Key. These files can either be obtained with IRL dollars from a Certificate Authroity or generated with OpenSSL like so:

openssl req -newkey rsa:2048 -new -x509 -days 3650 -nodes -out mongodb-cert.crt -keyout mongodb-cert.key
cat mongodb-cert.key mongodb-cert.crt > mongodb.pem

mongodb.pem will be used as the PEM file, mongodb-cert.key is the Private Key file, and mongodb-cert.crt is Certificate file which can also be used as the CA file. YOU WILL NEED ALL THREE OF THESE.

Step 3: Configure MongoD

We're going to assume that you copied these files to your /etc/ssl/ folder where they belong. Now we open up our MongoDB config file:

sudo vi /etc/mongod.conf

and modify the "# network interfaces" section like so:

# network interfaces
net:
  port: 27017
  #bindIp: 127.0.0.1
  ssl:
    mode: allowSSL
    PEMKeyFile: /etc/ssl/mongodb.pem
    #CAFile: /etc/ssl/mongodb-cert.crt

PLEASE NOTE: we are commenting out bindIp. THIS ALLOWS EXTERNAL CONNECTIONS to access your Mongo database. We assume that this is your end goal (Why would your encrypt traffic on localhost?), but you should only do this AFTER SETTING UP AUTHORIZATION RULES for your MongoDB server.

The CAFile is also commented out as it is optional. I will explain how to set up Certificate Authority trust at the end of this post.

As always, you must restart MongoDB before config file changes will take effect:

sudo service mongod restart

DID YOUR SERVER FAIL TO START? You are on your own, but there is probably an issue with your certificate files. You can check start-up errors by running mongod manually:

sudo mongod --config /etc/mongod.conf

Step 4: Test your server settings

Before we go messing with Node configurations, let's make sure that your server setup is working properly by connecting with the mongo command line client:

mongo --ssl --sslAllowInvalidHostnames --sslAllowInvalidCertificates

Unless the domain name on your certificate is 127.0.0.1 or localhost, the --sslAllowInvalidHostnames flag is necessary. Without it, you will probably get this error:

E NETWORK  The server certificate does not match the host name 127.0.0.1
E QUERY    Error: socket exception [CONNECT_ERROR] for 
    at connect (src/mongo/shell/mongo.js:179:14)
    at (connect):1:6 at src/mongo/shell/mongo.js:179
exception: connect failed

Step 5) Configure Node.JS / Mongoose

If you are using the node-mongodb-native package in your Node application, stop immediately and start using Mongoose. It's not that hard. That said, mongoose.connect() has virtually the same API as mongodb.connect(), so substitute appropriately.

var fs = require('fs')
  , mongoose = require('mongoose')
  , mongoUri = "mongodb://127.0.0.1:27017?ssl=true"
  , mongoOpt = {
      "server": { 
        "sslValidate": false,
        "sslKey": fs.readFileSync('/etc/ssl/mongodb.pem'),
        "sslCert": fs.readFileSync('/etc/ssl/mongodb-cert.crt')
      }
    }
  ;

mongoose.connect(mongoUri, mongoOpt);

Step 6) [Optionally] verify your Certificates via a Certificate Authority

In order to validate your SSL Certificates, you need to obtain a CA (or bundle) file from your Certificate Authority. This will look a lot like your Certificate file, but will often contain multiple Certificates (which form achain of trust to verify that a certificate is valid). If you are using a self-signed certificate, you can use your mongodb-cert.crt as a CA file.

You will also need to ensure that your MongoDB server's hostname matches the one used to create the certificate.

Step 6.3) Update your mongod configuration

sudo vi /etc/mongod.conf

and modify the "# network interfaces" section like so:

# network interfaces
net:
  port: 27017
  #bindIp: 127.0.0.1
  ssl:
    mode: allowSSL
    PEMKeyFile: /etc/ssl/mongodb.pem
    CAFile: /etc/ssl/mongodb-ca.crt

sudo service mongod restart

Step 6.4) Test your server settings

mongo --ssl --sslAllowInvalidHostnames --sslCAFile /etc/ssl/mongodb-ca.crt --sslPEMKeyFile /etc/ssl/mongodb.pem

Mongo Clients can pass in the CA file as well to verify that they are talking to the correct server. This is done with the --sslCAFile parameter

Mongo Servers configured with a CAFile require that clients possess a valid Certificate AND the Private Key for the server. In the mongo shell client, this is done by passing in the --sslPEMKeyFile parameter.

Without a PEM file (which contains the server's Certificate), you may see this error:

I NETWORK  DBClientCursor::init call() failed
E QUERY    Error: DBClientBase::findN: transport error: 127.0.0.1:27017 ns: admin.$cmd query: { whatsmyuri: 1 }
    at connect (src/mongo/shell/mongo.js:179:14)
    at (connect):1:6 at src/mongo/shell/mongo.js:179
exception: connect failed

The server can be configured to accept requests from clients without a PEM file by enabling net.ssl.weakCertificateValidation, but you will be weakening your security for no real gain.

Step 6.5) Configure Node.JS / Mongoose

There are a couple of gotchas here, so bare with me.

First, you NEED to have node-mongodb-native 2.0 or later. If you are using Mongoose, then you NEED Mongoose 4.0 or later. Previous Mongoose versions use node-mongodb-native 1.* which does not support Certificate validation in any capacity whatsoever.

Secondly, there is no sslAllowInvalidHostnames or similar option available in node-mongodb-native. This is not something that node-mongodb-native developers can fix (I would have by now) because the native TLS library available in Node 0.10.* offers no option for this. In Node 4.* and 5.*, there is a checkServerIdentity option which offers hope, but switching from the original Node branch to the branch after the io.js merge can cause a bit of headache at the current time.

So let's try this:

var fs = require('fs')
  , mongoose = require('mongoose')
  , mongoUri = "mongodb://127.0.0.1:27017?ssl=true"
  , mongoOpt = {
      "server": { 
        "sslKey": fs.readFileSync('/etc/ssl/mongodb.pem'),
        "sslCert": fs.readFileSync('/etc/ssl/mongodb-cert.crt'),
        "sslCa": fs.readFileSync('/etc/ssl/mongodb-ca.crt')
      }
    }
  ;

If you are getting hostname/IP mismatch errors, either fix your certificate, or negate all this hard work by disabling sslValidate:

var fs = require('fs')
  , mongoose = require('mongoose')
  , mongoUri = "mongodb://127.0.0.1:27017?ssl=true"
  , mongoOpt = {
      "server": {
        "sslValidate": false,
        "sslKey": fs.readFileSync('/etc/ssl/mongodb.pem'),
        "sslCert": fs.readFileSync('/etc/ssl/mongodb-cert.crt'),
        "sslCa": fs.readFileSync('/etc/ssl/mongodb-ca.crt')
      }
    }
  ;

Well I hope this helps someone. The information for getting MongoDB SSL working is out there, but it's all over the place. I did my best to assemble it into one document for you and future me.

Prevent less-css and less-middleware from modifying calc() attribute equations

So lets say you want to include a random third-party library into your web application such as Angular Material. This third-party library uses SCSS or SASS, but you are using LESS, so you do the only sensible thing, and import this library's CSS file into your LESS file so that you can use the library's classes as a base for future modification and inheritance by doing something like this:

@import (less) "../js/lib/angular/angular-material.min.css";

Which seems to work fine except for a few cases where some elements aren't aligned where you expect them to be (eg: the carets aren't all the way to the right side of your select fields). Upon deeper inspection you see that LESS is overwriting your multi-unit calc() calls. Now what originally was:

max-width: calc(100% - 2*8px);

appears as:

max-width: calc(84%);

and you're rightfully pissed off about it. Some guy on stackoverflow.com tells you to use escaped strings, but this is a third party library, and you don't want to go off tinkering with it in case you decide to upgrade it later. So what do you do?

The solution is pretty simple. Just pass the option  strictMath: true  to LESS and it will leave your equations alone. To me, this option is counter intuitive, and I would expect setting strictMath to 'false' would prevent LESS from diddling with my equations, but the reverse is true.

"But I'm using the less-middleware library in my node application, and I can't figure out how to pass arbitrary LESS options to it." 

As of version 2.0.0, less-middleware accepts a 'render' option, which takes a plain object as a list of lower-level LESS options. The option is called 'render' because this object is passed directly to the less.render() function within the middleware. I find the naming choice here a bit confusing, and it doesn't make sense to anyone who is just reading the documentation and hasn't opened up the less-middleware source code yet. Anyways, here it is in action:

var lessMiddleware = require('less-middleware');
var lessPath = path.join(__dirname, '/public')
...
app.use(lessMiddleware(lessPath, { render: { strictMath: true }}));

That wasn't so hard, was it? Well screw yourself, because I spent over an hour figuring this out. Hopefully this post will stop that from happening to anyone else.

How to Remove control options from NVD3.js Stacked Area Chart (Stacked, Streamed, or Expanded view)

I recently came across a situation where one of my clients had no need for the 'Expanded' view mode in the NVD3 stacked area chart that I had customized for them. The documentation for NVD3 is often lacking, but the source code is well organized and makes discovering hidden options fairly quick.

To remove all of the controls, you can simple call:
chart.showControls(false);
To remove a single control, you have to set the chart._options.controlOptions variable to set the controls that you want to show. The default options are:
['Stacked', 'Stream', 'Expanded']

So you if need to remove one of the control options, set chart._options.controlOptions to an array with that option omitted:
chart._options.controlOptions = ['Stacked', 'Stream']; // hide 'Expanded' view

chart._options.controlOptions = ['Stacked', 'Expanded']; // hide 'Stream' view

chart._options.controlOptions = ['Expanded', 'Stream']; // hide 'Stacked' view
The chart will default to the first option, so in the last example, 'Expanded' will be the default view. If the order of controls is switched, 'Stream' will be the default view.

Running Raw/Direct/Native MongoDB Queries Through Mongoose.js

Sometimes it may be necessary (particular when modifying a schema), to bypass mongoose schema constraints and access a mongoose database through the native mongodb driver.
Generally, this is as simple as accessing Model.collection 
However, there is a caveat when running find() queries. All native driver documentation suggests that Model.collection.find({}) should return a cursor, but this is not the case with Mongoose.
Running Model.collection.find({}).toArray(function(err, data){}) will throw an error
TypeError: Cannot call method 'toArray' of undefined
because find() will return undefined
Instead, you must pass a callback to find() which accepts the cursor as the second argument. Here is an example:
Model.collection.find({}, function(err, cursor) {
cursor.toArray(function(err, data){
// process data array
})
})

Taking a PNG Screenshot of a Webpage from PHP + Apache on Ubuntu Server

While it may not be terrible useful, I decided that I had to know how to generate a screenshot of a webpage from my own webpage. Please note that THIS OPENS EXTRA SECURITY RISKS and SHOULD ONLY BE DONE ON AN ISOLATED SERVER where NOTHING OF VALUE CAN BE LOST

** If this is running on Ubuntu Server, obviously you need to install gnome to make this work:
sudo apt-get ubuntu-desktop
sudo shutdown -r now
export DISPLAY=:0.0

Install gnome-web-photo
sudo apt-get install gnome-web-photo

Login with www-data
sudo su www-data

** If you receive this error:
This account is currently not available.
You need to recreate user www-data:
service apache2 stop
deluser www-data
delgroup www-data
adduser www-data
service apache2 restart

Try to run gnome-web-photo
gnome-web-photo http://google.com ~/test.png
No protocol specified

** (gnome-web-photo:13000): WARNING **: Could not open X display

No protocol specified
Cannot open display: 
Usage: gnome-web-photo [--mode=photo|thumbnail|print] [...]

It looks like our little friend apache cannot connect to our xserver. That's an easy fix, we just need to add user www-data to the privileged users who can access the active display.

With an account that can access the display:
xhost +SI:localuser:www-data

With www-data account:
export DISPLAY=":0.0"
gnome-web-photo http://google.com ~/test.png

It should work now! You can use exec() and shell_exec() to now take browser shots from a PHP page. Go ahead, run your own browershots website.
exec("export DISPLAY=:0.0;gnome-web-photo http://google.com $dir/fuck.png");


Take a Browser Screen Shot of a Webpage and Convert it to a PNG Image in Ubuntu 13.10

Before I start ranting about all of the impossible ways that people claim should work, here is actual solution:

sudo apt-get install gnome-web-photo
gnome-web-photo http://google.com gnome-web-photo.png

Why was that so difficult, internet? Why so many options that do not work? I blame the google algorithms that place value on the oldest links. The oldest method is seldom the best when technology is concerned.

If the above doesn't work, here is a backup method:

sudo apt-get install wkhtmltopdf imagemagick
wkhtmltopdf www.google.com google.pdf
convert -quality 100 google.pdf google.jpg


Now let's talk about about the stuff that doesn't work.

wkhtmltoimage
Does this work on your OS? I just get seg faults.

khtml2png
This hasn't been hip in 4+ years and clearly doesn't run on modern Ubuntu installations.

webkit2png and PyWebShot
Both of these supposedly work great... if you're on a Mac. Why are you on a Mac? I have no idea. Does anyone run a Mac server? No. So what is the point of programatically taking a browser shot on them anyways? Who cares.
Apparently Apple bought some company that owned some code or something that webkit python bindings need or whatever who cares. The short story is that you cannot install pyobjc on Linux because of Nazism because you get some horrible pip error:
ValueError: invalid literal for int() with base 10: ''
So basically, it's looking for your OSX version but can't find it because you aren't stupid enough to be using a Mac. Every day I get more glad Steve Jobs died an untimely death. What a prick.

Full Text Searching InnoDB Tables on MySQL 5.5 with Trigger-based Table Syncing

As you may be aware, MySQL 5.6 supports full text searches on InnoDB tables, but few are willing to upgrade yet because 5.6 is currently unstable and not yet suitable for production servers. This means that the only way to use full text searching in MySQL is to put this data into a MyISAM table. Unfortunately MyISAM tables suffer from a number of faults, the most worrisome being data integrity.

The solution for me is to use InnoDB tables for everything, but duplicate the searchable data into a MyISAM table. I then use MySQL triggers to keep the MyISAM table's data automatically synced up with the InnoDB. If the MyISAM table gets corrupted, I can wipe it and copy the data back over with extreme ease.

To demonstrate, assume that I have a table called `links` with all sorts of columns and rows in it. Of these columns, I want the title and description to be searchable.

The first step is to create a separate MyISAM table to store the searchable data. Make sure the column definitions are the same, but only include the primary key and the rows you want to search.

CREATE TABLE `links_searchable` (
    `id` int(10) unsigned NOT NULL,
    `title` varchar(100) NOT NULL,
    `description` varchar(1000) NOT NULL
) ENGINE=MyISAM;

The next step is to create a fulltext index on the searchable columns, so that MySQL will allow the MATCH() queries to be executed on these columns.

create fulltext index `ft` on links_searchable (title,description);

If your `links` table has any data in it, go ahead and copy it over now

insert links_searchable (id,description,title) select id,description,title from links;

Now create the MySQL triggers so that you don't have to write extra code to keep these tables synced. Once these triggers are created, you can forget about this extra table until it's actually time to search.

CREATE TRIGGER ins_searchable AFTER INSERT ON `links`
    FOR EACH ROW
        insert into links_searchable (id,description,title) values(NEW.id,NEW.description,NEW.title);

CREATE TRIGGER del_searchable AFTER DELETE ON `links`
FOR EACH ROW
DELETE FROM links_searchable WHERE id=old.id;

CREATE TRIGGER upd_searchable AFTER UPDATE ON `links`
FOR EACH ROW
    UPDATE links_searchable SET title=NEW.title, description=NEW.description WHERE id=NEW.id;

At this point, you should be able to run full text searches on your data. Here is a MATCH() query that returns all of the relevant links data. The results are automatically sorted by relevance.

select links.* from links, links_searchable where MATCH (links_searchable.title,links_searchable.description) AGAINST ('query') and links.id=links_searchable.id

Now, if you are like me, you hate stopwords. Disabling them is pretty easy in Ubuntu. Simply add or update these lines in your /etc/mysql/my.cnf under [mysqld]

ft_stopword_file = ""
ft_min_word_len = 1

Then restart MySQL

sudo service mysql restart

And finally, reindex your searchable table

repair table links_searchable;

It's that easy. Enjoy all of the benefits of both MyISAM and InnoDB by using both at the same time :)

Install Android SDK on Ubuntu 13.10

Download and unzip to ~/adt
cd ~
wget http://dl.google.com/android/adt/adt-bundle-linux-x86-20140702.zip
unzip adt-bundle-linux-x86-20140702.zip
rm adt-bundle-linux-x86-20140702.zip
mv adt-bundle-linux-x86-20140702 adt

Install dependencies
sudo apt-get install openjdk-7-jre openjdk-7-jdk

Include the /sdk/tools and /sdk/platform-tools folders in your environment path
echo 'export PATH=${PATH}:~/adt/sdk/tools'|cat - ~/.bashrc > /tmp/out && mv /tmp/out ~/.bashrc
echo 'export PATH=${PATH}:~/adt/sdk/platform-tools'|cat - ~/.bashrc > /tmp/out && mv /tmp/out ~/.bashrc
source ~/.bashrc

run the /sdk/tools/android binary and install updates
android

run the android emulator to make sure everything is working
android avd