Unreal Engine 4 pointer cursor stuck on web browser widget

I know UE4's web browser widget is an "experimental" feature, but this is a pretty obnoxious bug that they have refused to fix for at least 4 years.

As far as I can tell, as soon as your cursor enters the "pointer" state while inside the browser widget, it will be stuck there indefinitely, even after you close the widget. The only solution I have found is to prevent it from ever reaching that state.

If you have full control over the webpage you are displaying, you can modify your CSS to use some other cursor type for all your elements. I did not.

My solution? Set the cursor for every element to 'default' using javascript.
var nodes = document.querySelectorAll('*');
for(var i=0; i<nodes.length; i++) {
    nodes[i].setAttribute('style', 'cursor:default !important'); 
}

When? All of the time.

It does seem excessive to do every element every tick, and it probably is, but when you have no control over a webpage, you never know when some new pesky element is going to be shoved into the DOM and screw you again. Why risk it?

One thing I did try was to set a software "Hand" cursor. I assume this translates to cursor:pointer, but who knows? That did nothing at all. Maybe the bug here is that there is no "pointer" cursor type in UE4, so the web browser widget has no idea how to turn it off.

One thing I did NOT try was to set up invisible hardware cursor images. Perhaps there is a conflict in the transition from hardware to software cursors? I do not know, but I do know that I end up with both when I mouse out of an HTML button. Forever.


Reducing Wordpress, PHP, MySQL Memory Usage

BACKGROUND


I hate WordPress, but I also love WordPress. It is still the most user-friendly open source blogging platform in existence. I have tried many alternatives, I've even written my own, but the feature set has always felt lacking.

My main problem with WordPress used to be the codebase. It was a mess, but the devs have managed to clean it up quite a bit. The current version seems quite tidy as far as PHP projects go, and hacking on it isn't nearly as frustrating as it had been in earlier versions.

So now my big beef is with performance. PHP and MySQL are slow, gobble up tons of resources even when sitting idle, and WordPress' extensive feature set dumps a ton of bloat on top.

At the end of the day, I just want users to load my dumb blog as fast as possible and have WordPress leave as small of a footprint as possible so I can save my server resources for more interesting projects.


HOW TO REDUCE MYSQL MEMORY USAGE


The first thing I notice when checking my system memory is MySQL. Why are you eating half of my memory, MySQL? You're not doing anything. Just give it back.

MySQL loves to cache. It increases query speed tremendously to store your entire database in memory, but that's not my goal. Once I'm done writing my post, I won't need MySQL anymore, because I will be generating a static version of my blog for public viewing. Nothing serves faster than static files, and I intend my blog to be read only for the general public.

My end goal now is to dial down MySQL's memory usage as far as possible while still being able to use the Wordpress admin panel without noticeable lag.

So open up your MySQL config (mine was at /etc/mysql/mysql.conf.d/mysqld.cnf) and start tweaking some settings.

The biggest single improvement that I was able to make was to turn off MySQL's Performance Schema by adding this line:

performance_schema = 0

I also found a huge list of variables here and then proceeded to completely ignore the "minimum" values:

key_buffer_size = 8
max_allowed_packet = 1K
max_connections = 10
query_cache_limit = 0
query_cache_size = 0
expire_logs_days = 1
max_binlog_size = 1K
thread_stack = 1
thread_cache_size = 0
host_cache_size = 0
innodb_buffer_pool_size = 0
innodb_log_buffer_size = 0
innodb_ft_cache_size = 0
innodb_ft_total_cache_size = 0
sort_buffer_size = 0
read_buffer_size = 0
read_rnd_buffer_size = 0
max_heap_table_size = 0
tmp_table_size = 0
bulk_insert_buffer_size = 0
join_buffer_size = 0
net_buffer_length = 0
innodb_sort_buffer_size = 0
binlog_cache_size = 0
binlog_stmt_cache_size = 0

Now just save and restart MySQL.

I'm amazed that WordPress can still function at all with these settings, but honestly I don't even notice the difference when clicking around the admin panel. But what I did notice what that I had slashed MySQL's memory usage by 90%.


HOW TO REDUCE PHP MEMORY USAGE


Just like MySQL, I will only need PHP to write posts, not to serve them. Again, I will be minimizing PHP's memory usage by modifying configuration files.

PHP's biggest issue is that, by default, it spawns multiple processes to handle incoming requests, and then refuses to kill them once the requests have been handled. These processes, even when sitting idle, take up way too much of my precious memory.

So most of the work can be done in the pool config. For my system, that file is at /etc/php/7.0/fpm/pool.d/www.conf

pm = ondemand
pm.process_idle_timeout = 1s

This configuration tells PHP to not create any listener processes until a request is actually made. If a process does nothing for more than 1 second, it will be killed. Humanely. (PETA plz no bully.)

So PHP can now eat all the memory it wants when I'm actually using it (almost never) and will give it back immediately after. If you are worried about peak memory usage, you can set 

pm.max_children = 1

Other than than you can't do much. Limiting a PHP thread's max memory size will result in errors. You are essentially at the mercy of WordPress' dev team.


HOW TO REDUCE WORDPRESS MEMORY USAGE


All of this is has been a lead up to the main objective, building WordPress into a static website. Static websites are faster, more stable, and more secure. They also require very little memory to serve.

If you don't mind manually generating your public-facing WordPress site, I would recommend installing the Simply Static WordPress Plugin, and just use that.

If you're lazy, I'll share how I was able to trigger a static site generation every time a post is published or updated.

The first step is to create a shell script to clone your WordPress site into a public directory. I called mine static.sh

# Download your dynamic wordpress site into /tmp
wget --trust-server-names --no-clobber --mirror --convert-links --adjust-extension -e robots=off -P /tmp http://privateblog.com

# Search for any mention of your private domain and replace it with your public domain
find /tmp/privateblog.com -type f -print0 | xargs -0 sed -i 's/privateblog\.com/publicblog\.com/g'

# Manually copy your uploads directory to make sure you have all the images sizes that wget ignored
cp -r /var/www/privateblog.com/wp-content/uploads/* /tmp/privateblog.com/wp-content/uploads/

# Move the result into your public www directory

rm -rf /var/www/publicblog/*
mv /tmp/privateblog.com/* /var/www/publicblog/

Now we have a bash script that needs to run whenever we update our blog. Luckily WordPress makes this easy by shipping with XML-RPC functionality.

Since XML-RPC uses HTTP as a transport protocol, we don't need any extra technology to detect a change; we can just use PHP. Of course, if you wanted to analyze the XML payload, you would need to set up an actual XML-RPC server, but that's not necessary here.

So now we just need a PHP script added to our WordPress directory which calls static.sh. I called mine rpc.php

<?php passthru('./static.sh'); ?>


Nice. The last piece of this is to add the URL of rpc.php to our WordPress Update Services. In the dashboard, go to Settings > Writing and find Update Services down at the bottom. Add your URL to the box, and hit Save Change.


That should do the trick. If you run into problems, double check your file permissions and ownership. Keep in mind that the www-data user needs to be able to execute both scripts and have write access to your public directory.

Disable $mdBottomSheet Drag Events

I recently ran into a situation where I needed to prevent an md-bottom-sheet component from dragging around. Inside the bottom sheet, we have an image cropper component (ui-cropper) which has it's own drag events. When we drag the cropper window up, the entire bottom sheet moves up with it. When we drag the cropper window down, the bottom sheet closes.


Obviously a bottom sheet isn't the best element to contain an image cropper, but hey, I try to avoid arguments with my UX team when possible.

Now because the version of Angular Material we are using doesn't support drag disabling for md-bottom-sheet, we have to hack around it. I'm okay with a little bit of hacking, but I'm certainly not going to modify Angular Material's core files over a questionable UX decision.

My compromise is to add a bit of code to our directive which intercepts and neutralizes $mdDrag events. Five lines of code for a one-off conflict doesn't seem like the end of the world.

The directive I'm modifying is nothing more than a button which opens a bottom sheet with a small form in it. All I need to do is attach a short function disableDrag to the scope which would is triggered when the bottom sheet is clicked.

function link(scope) {

  scope.disableDrag = disableDrag;
  scope.onClick = onClick;
    
  function disableDrag() {
    $('md-bottom-sheet') // find the bottom element
      .on('$md.drag', e => false) // hold it still
      .on('$md.dragend', e => false); // prevent closing
  }
    
  function onClick() {
    $mdBottomSheet.show({ 
      templateUrl: require('./bottomSheet.html') 
    }).then(() => {
      // process form data
    });
  }
}

Now all I had to do was link it to the bottom sheet element.

<md-bottom-sheet ng-mousedown='disableDrag()'>
  <!-- our form -->
</md-bottom-sheet>

Works like a charm.

Stop SVGO and GULP-SVGMIN from Breaking Paths and Colors in SVG Images

When practical, I like to use vector images on my websites. They are great for responsive design, and look much more clean and crisp than PNG or JPG images. Of course, I am also a fan of performance, and SVG files can often come significantly larger than they need to be.

For JPG and PNG images, I use a tool called Trimage to compress them. For SVG files, I use SVGO. Actually, I use the gulp package gulp-svgmin, but that is just a gulp wrapper for SVGO.

The problem with SVGO, is that with the default settings, it tends to break many SVG images that come out of tools like Adobe Illustrator. For your benefit, I have trial-and-error'd my way through all of the SVGO plugins and isolated the options which cause issues.

If your SVG image paths are breaking, turn off convertPathData and mergePaths.

If your colors are turning black or being removed, turn off removeUnknownsAndDefaults.

There have also been reports of issues with collapseGroups, but I have not had an issue yet.

Here is a snippet from the gulp file I am currently using:
var gulp = require('gulp');
var svgmin = require('gulp-svgmin');
var debug = require('gulp-debug');

gulp.task('compress', function () {
  return gulp.src('./public/img/**/*.svg')
    .pipe(debug())
    .pipe(svgmin({plugins: [
      { convertPathData: false } // breaks paths
      , { mergePaths: false } // breaks paths
      , { removeUnknownsAndDefaults: false } // breaks colors
    ]}))
    .pipe(gulp.dest('./public/img/'));
});

SVG image files may still be larger than PNG counterparts, but I find that using a gzip compression tool like the nodejs compression package can often crunch these optimized SVG files another 300%, making them significantly smaller than gzipped PNG files, and they look so much better!

Securing MongoDB with Authentication and Authorization using Node.JS and Mongoose on Ubuntu

One of the nice things about MongoDB is that it doesn't force your to deal with account management; you simply install it and you can immediately start cramming it full of data anonymously.

But if you need to access a database remotely, or you just want an extra layer of security for your data, setting up MongoDB user accounts and restricting access is very simple and highly recommended.

Step 1) Create an administrator account

It's a swell idea to create an account with access to all databases, and then use this account to create more restricted accounts for users and applications.

Each user is stored in and authenticated against a single database, though you can grant it access to other databases using the 'roles' attribute. When creating a user, it will be created in the current database, so let's switch to the 'admin' database to store our admin user in:

use admin;

To create a user in the currently selected database, we call db.createUser() and specify 'user', 'pwd', and 'roles' attributes. To create a root user (one with full privileges across all databases) named 'admin' with password 'pass':

db.createUser({
  user : "admin",
  pwd  : "pass",
  roles: [
    { 
      role: 'root', db: 'admin'
    }
  ]
});

You can see that each role object in the roles array requires a role name and a database name. For some reason, you still have to specify a database even if the role allows access to all databases.

There are a ton of Built-in MongoDB user roles, and they will meet most of your needs. If they do not, please be aware that you can create your own custom user roles. Useful roles include 'read', 'readWrite', 'dbAdmin', 'userAdmin', 'backup', 'restore', and 'root'

Step 2) Test your administrator account

mongo admin -u admin -p pass

Even though this user has full access to all databases, authentication will fail if we do not connect to the admin database. This is because the admin user is stored in the admin database, and mongo needs to be told which database the user is stored in before it can authenticate them. By default, mongo assumes the user is stored in the database that you are connecting to, but you can override this assumption by using the --authenticationDatabase flag

mongo -u admin -p pass --authenticationDatabase admin

Step 3) Create a restricted user for your application

It's not the most wise strategy to use your root account for your application. Each user and application should have their own account that is restricted to access only the data that they need to function.

For this we will assume that you have a database named 'app' that you use for your web application, and your application needs to read and write to 'app'.

use app;

db.createUser({
  user : "appuser",
  pwd  : "apppass",
  roles: [
    { 
      role: 'readWrite', db: 'app'
    }
  ]
});

Step 4) Force authorization

Right now, your mongo database accepts authentication, but authentication is not required. In order to force users to authenticate, we have to enable it in the /etc/mongod.conf configuration file

FOR MONGODB 3.0 AND ABOVE add these lines:

security:
  authorization: enabled

FOR MONGODB 2.X add this line:

auth = true

Any changes to your config file require a restart before they will take effect:

sudo service mongod restart

Step 5) Connect with Node.js / Mongoose.js

var fs = require('fs')
  , mongoose = require('mongoose')
  , mongoUri = "mongodb://appuser:apppass@127.0.0.1:27017/app"
  ;

mongoose.connect(mongoUri);

All of the magic here is contained in the mongo connection string. This same connection string also works with the native mongodb driver.

Here we are connecting the same database that the user is stored in (app), so it should authenticate properly. To connect to a different database that the user is not stored in, we can add the 'authSource' attribute. Here we are connecting to the 'app2' database with 'appuser' who is stored in the 'app' database:

mongodb://appuser:apppass@127.0.0.1:27017/app2?authSource=app"

Step 6) [Optionally] allow external connections

If you need to connect to your database from another machine in the network, you have to tell mongo to bind to an external IP address.

FOR MONGODB 3.0 AND ABOVE comment the bindIp line:

net:
  # bindIp: 127.0.0.1

FOR MONGODB 2.X comment the bind_ip line:

# bind_ip = 127.0.0.1

And restart:

sudo service mongod restart

Step 7) [Optionally] encrypt your MongoDB traffic with TLS/SSL

Connecting to MongoDB over TLS/SSL with Node.JS and Mongoose on Ubuntu

Step 1: Obtain MongoDB 3.0

The first thing you need to know is that that SSL is only supported out-of-the-box by MongoDB 3.0 and later. Ubuntu does not have 3.0 in the default repositories, so here's how you get it:

sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 7F0CEB10
echo "deb http://repo.mongodb.org/apt/ubuntu trusty/mongodb-org/3.0 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-3.0.list
sudo apt-get update
sudo apt-get install -y mongodb-org=3.0.7 mongodb-org-server=3.0.7 mongodb-org-shell=3.0.7 mongodb-org-mongos=3.0.7 mongodb-org-tools=3.0.7

3.0.7 is the latest stable version as of now, but feel free to substitute 3.0.7 with your favorite release.

Step 2: Obtain Private Key, Certificate, and PEM files

The PEM contains a Public Key Certificate and its associated Private Key. These files can either be obtained with IRL dollars from a Certificate Authroity or generated with OpenSSL like so:

openssl req -newkey rsa:2048 -new -x509 -days 3650 -nodes -out mongodb-cert.crt -keyout mongodb-cert.key
cat mongodb-cert.key mongodb-cert.crt > mongodb.pem

mongodb.pem will be used as the PEM file, mongodb-cert.key is the Private Key file, and mongodb-cert.crt is Certificate file which can also be used as the CA file. YOU WILL NEED ALL THREE OF THESE.

Step 3: Configure MongoD

We're going to assume that you copied these files to your /etc/ssl/ folder where they belong. Now we open up our MongoDB config file:

sudo vi /etc/mongod.conf

and modify the "# network interfaces" section like so:

# network interfaces
net:
  port: 27017
  #bindIp: 127.0.0.1
  ssl:
    mode: allowSSL
    PEMKeyFile: /etc/ssl/mongodb.pem
    #CAFile: /etc/ssl/mongodb-cert.crt

PLEASE NOTE: we are commenting out bindIp. THIS ALLOWS EXTERNAL CONNECTIONS to access your Mongo database. We assume that this is your end goal (Why would your encrypt traffic on localhost?), but you should only do this AFTER SETTING UP AUTHORIZATION RULES for your MongoDB server.

The CAFile is also commented out as it is optional. I will explain how to set up Certificate Authority trust at the end of this post.

As always, you must restart MongoDB before config file changes will take effect:

sudo service mongod restart

DID YOUR SERVER FAIL TO START? You are on your own, but there is probably an issue with your certificate files. You can check start-up errors by running mongod manually:

sudo mongod --config /etc/mongod.conf

Step 4: Test your server settings

Before we go messing with Node configurations, let's make sure that your server setup is working properly by connecting with the mongo command line client:

mongo --ssl --sslAllowInvalidHostnames --sslAllowInvalidCertificates

Unless the domain name on your certificate is 127.0.0.1 or localhost, the --sslAllowInvalidHostnames flag is necessary. Without it, you will probably get this error:

E NETWORK  The server certificate does not match the host name 127.0.0.1
E QUERY    Error: socket exception [CONNECT_ERROR] for 
    at connect (src/mongo/shell/mongo.js:179:14)
    at (connect):1:6 at src/mongo/shell/mongo.js:179
exception: connect failed

Step 5) Configure Node.JS / Mongoose

If you are using the node-mongodb-native package in your Node application, stop immediately and start using Mongoose. It's not that hard. That said, mongoose.connect() has virtually the same API as mongodb.connect(), so substitute appropriately.

var fs = require('fs')
  , mongoose = require('mongoose')
  , mongoUri = "mongodb://127.0.0.1:27017?ssl=true"
  , mongoOpt = {
      "server": { 
        "sslValidate": false,
        "sslKey": fs.readFileSync('/etc/ssl/mongodb.pem'),
        "sslCert": fs.readFileSync('/etc/ssl/mongodb-cert.crt')
      }
    }
  ;

mongoose.connect(mongoUri, mongoOpt);

Step 6) [Optionally] verify your Certificates via a Certificate Authority

In order to validate your SSL Certificates, you need to obtain a CA (or bundle) file from your Certificate Authority. This will look a lot like your Certificate file, but will often contain multiple Certificates (which form achain of trust to verify that a certificate is valid). If you are using a self-signed certificate, you can use your mongodb-cert.crt as a CA file.

You will also need to ensure that your MongoDB server's hostname matches the one used to create the certificate.

Step 6.3) Update your mongod configuration

sudo vi /etc/mongod.conf

and modify the "# network interfaces" section like so:

# network interfaces
net:
  port: 27017
  #bindIp: 127.0.0.1
  ssl:
    mode: allowSSL
    PEMKeyFile: /etc/ssl/mongodb.pem
    CAFile: /etc/ssl/mongodb-ca.crt

sudo service mongod restart

Step 6.4) Test your server settings

mongo --ssl --sslAllowInvalidHostnames --sslCAFile /etc/ssl/mongodb-ca.crt --sslPEMKeyFile /etc/ssl/mongodb.pem

Mongo Clients can pass in the CA file as well to verify that they are talking to the correct server. This is done with the --sslCAFile parameter

Mongo Servers configured with a CAFile require that clients possess a valid Certificate AND the Private Key for the server. In the mongo shell client, this is done by passing in the --sslPEMKeyFile parameter.

Without a PEM file (which contains the server's Certificate), you may see this error:

I NETWORK  DBClientCursor::init call() failed
E QUERY    Error: DBClientBase::findN: transport error: 127.0.0.1:27017 ns: admin.$cmd query: { whatsmyuri: 1 }
    at connect (src/mongo/shell/mongo.js:179:14)
    at (connect):1:6 at src/mongo/shell/mongo.js:179
exception: connect failed

The server can be configured to accept requests from clients without a PEM file by enabling net.ssl.weakCertificateValidation, but you will be weakening your security for no real gain.

Step 6.5) Configure Node.JS / Mongoose

There are a couple of gotchas here, so bare with me.

First, you NEED to have node-mongodb-native 2.0 or later. If you are using Mongoose, then you NEED Mongoose 4.0 or later. Previous Mongoose versions use node-mongodb-native 1.* which does not support Certificate validation in any capacity whatsoever.

Secondly, there is no sslAllowInvalidHostnames or similar option available in node-mongodb-native. This is not something that node-mongodb-native developers can fix (I would have by now) because the native TLS library available in Node 0.10.* offers no option for this. In Node 4.* and 5.*, there is a checkServerIdentity option which offers hope, but switching from the original Node branch to the branch after the io.js merge can cause a bit of headache at the current time.

So let's try this:

var fs = require('fs')
  , mongoose = require('mongoose')
  , mongoUri = "mongodb://127.0.0.1:27017?ssl=true"
  , mongoOpt = {
      "server": { 
        "sslKey": fs.readFileSync('/etc/ssl/mongodb.pem'),
        "sslCert": fs.readFileSync('/etc/ssl/mongodb-cert.crt'),
        "sslCa": fs.readFileSync('/etc/ssl/mongodb-ca.crt')
      }
    }
  ;

If you are getting hostname/IP mismatch errors, either fix your certificate, or negate all this hard work by disabling sslValidate:

var fs = require('fs')
  , mongoose = require('mongoose')
  , mongoUri = "mongodb://127.0.0.1:27017?ssl=true"
  , mongoOpt = {
      "server": {
        "sslValidate": false,
        "sslKey": fs.readFileSync('/etc/ssl/mongodb.pem'),
        "sslCert": fs.readFileSync('/etc/ssl/mongodb-cert.crt'),
        "sslCa": fs.readFileSync('/etc/ssl/mongodb-ca.crt')
      }
    }
  ;

Well I hope this helps someone. The information for getting MongoDB SSL working is out there, but it's all over the place. I did my best to assemble it into one document for you and future me.

Prevent less-css and less-middleware from modifying calc() attribute equations

So lets say you want to include a random third-party library into your web application such as Angular Material. This third-party library uses SCSS or SASS, but you are using LESS, so you do the only sensible thing, and import this library's CSS file into your LESS file so that you can use the library's classes as a base for future modification and inheritance by doing something like this:

@import (less) "../js/lib/angular/angular-material.min.css";

Which seems to work fine except for a few cases where some elements aren't aligned where you expect them to be (eg: the carets aren't all the way to the right side of your select fields). Upon deeper inspection you see that LESS is overwriting your multi-unit calc() calls. Now what originally was:

max-width: calc(100% - 2*8px);

appears as:

max-width: calc(84%);

and you're rightfully pissed off about it. Some guy on stackoverflow.com tells you to use escaped strings, but this is a third party library, and you don't want to go off tinkering with it in case you decide to upgrade it later. So what do you do?

The solution is pretty simple. Just pass the option  strictMath: true  to LESS and it will leave your equations alone. To me, this option is counter intuitive, and I would expect setting strictMath to 'false' would prevent LESS from diddling with my equations, but the reverse is true.

"But I'm using the less-middleware library in my node application, and I can't figure out how to pass arbitrary LESS options to it." 

As of version 2.0.0, less-middleware accepts a 'render' option, which takes a plain object as a list of lower-level LESS options. The option is called 'render' because this object is passed directly to the less.render() function within the middleware. I find the naming choice here a bit confusing, and it doesn't make sense to anyone who is just reading the documentation and hasn't opened up the less-middleware source code yet. Anyways, here it is in action:

var lessMiddleware = require('less-middleware');
var lessPath = path.join(__dirname, '/public')
...
app.use(lessMiddleware(lessPath, { render: { strictMath: true }}));

That wasn't so hard, was it? Well screw yourself, because I spent over an hour figuring this out. Hopefully this post will stop that from happening to anyone else.