Prevent FFMPEG from removing KLV data stream from MPEG-TS output

ffmpeg is a great tool for recording live video streams to files, clipping video files, streaming video files to udp, and a bunch of stuff I don't care about. Of course, I feel like it's always a battle to get the right flags to do exactly what I need.

The most recent example, I just want to copy an MPEG-TS file:

ffmpeg -i ./CheyenneVAhospital.ts out.ts

Yeah, of course it isn't that easy. ffmpeg has decided that I have no need for the KLV data stream.

Input #0, mpegts, from './CheyenneVAhospital.ts':

  Duration: 00:01:05.15, start: 271.125322, bitrate: 5535 kb/s

  Program 1 

  Stream #0:0[0x100]: Video: h264 (Baseline) ([27][0][0][0] / 0x001B), yuv420p(progressive), 1920x1080 [SAR 1:1 DAR 16:9], 29.97 fps, 29.97 tbr, 90k tbn, 59.94 tbc

  Stream #0:1[0x101]: Audio: aac (LC) ([15][0][0][0] / 0x000F), 48000 Hz, stereo, fltp, 127 kb/s

  Stream #0:2[0x102]: Data: klv (KLVA / 0x41564C4B)

Stream mapping:

  Stream #0:0 -> #0:0 (h264 (native) -> mpeg2video (native))

  Stream #0:1 -> #0:1 (aac (native) -> mp2 (native))

Press [q] to stop, [?] for help

Spoiler: I do need it. Very much. And after trying many flags (codec:data, map_metadata, copy_unknown, etc.), I found that I had to explicitly map all three stream types.

ffmpeg -i ./CheyenneVAhospital.ts -map 0:v -map 0:a -map 0:d out.ts

Input #0, mpegts, from './CheyenneVAhospital.ts':

  Duration: 00:01:05.15, start: 271.125322, bitrate: 5535 kb/s

  Program 1 

  Stream #0:0[0x100]: Video: h264 (Baseline) ([27][0][0][0] / 0x001B), yuv420p(progressive), 1920x1080 [SAR 1:1 DAR 16:9], 29.97 fps, 29.97 tbr, 90k tbn, 59.94 tbc

  Stream #0:1[0x101]: Audio: aac (LC) ([15][0][0][0] / 0x000F), 48000 Hz, stereo, fltp, 127 kb/s

  Stream #0:2[0x102]: Data: klv (KLVA / 0x41564C4B)

File 'f.ts' already exists. Overwrite? [y/N] y

Stream mapping:

  Stream #0:0 -> #0:0 (h264 (native) -> mpeg2video (native))

  Stream #0:1 -> #0:1 (aac (native) -> mp2 (native))

  Stream #0:2 -> #0:2 (copy)

Press [q] to stop, [?] for help

Simple enough, but I am compelled by future me to preserve this knowledge in the cloud so that I can further delay defragmenting my brain.

Using multicat to send an MPEG-TS video file to a UDP multicast address

Multicat is fairly simple to use. Just give it an input and output and it figures the rest out on it's own. 

multicat -uU ./CheyenneVAhospital.ts 226.6.6.1:8209

The -uU flags to ignore the missing RTP headers.

But wait! You need a .aux file apparently

error: couldn't open file ./CheyenneVAhospital.aux (No such file or directory)

Piece of cake. The docs say you just have to ingest the file with the ingests command. Let's copy and paste that into our shell

ingests -p 68 ./CheyenneVAhospital.ts

Oh great. That didn't work.

debug: end of file reached

error: no PCR found

Alright our PCR PID must be wrong. Luckily we can use ffprobe to get the PCR PID out of a .ts file.

ffprobe CheyenneVAhospital.ts -show_programs | grep pcr_pid

...

pcr_pid=256

Oh lovely! Why can't ingests do that? I dunno man stop asking questions.

Let's try ingests with the correct PCR PID

ingests -p 256 ./CheyenneVAhospital.ts

Hey, cool. No errors this time. And our .aux file is more than 0 bytes!

Now we can finally do the thing.

multicat -uU ./CheyenneVAhospital.ts 226.6.6.1:8209

Oh by the way, if you want to change your MPEG-TS file to have a consistent video bitrate, you can use ffmpeg for that. Here's an example of converting the video stream to 1Mb/s

ffmpeg -i ./CheyenneVAhospital.ts -c:v libx264 -b:v 1M -map 0 CheyenneVAhospital_1M.ts

And multicat doesn't support looping, so you'll just have to put the multicat command inside a loop body and deal with the spikes

for (( ; ; ))

do

multicat -uU ./CheyenneVAhospital_1M.ts 226.6.6.1:8209

done

VBoxManage: error: The virtual machine 'minikube' has terminated unexpectedly

Full error: 

Exiting due to PR_VBOX_BLOCKED: Failed to start host: creating host: create: creating: Unable to start the VM: /usr/bin/VBoxManage startvm minikube --type headless failed:

VBoxManage: error: The virtual machine 'minikube' has terminated unexpectedly during startup with exit code 1 (0x1)

VBoxManage: error: Details: code NS_ERROR_FAILURE (0x80004005), component MachineWrap, interface IMachine

Solution:

# Purge VirtualBox.

sudo killall VirtualBox

sudo apt-get remove --purge virtualbox

rm -rf ~/.config/VirtualBox/

# Wipe minikube cache

rm -rf ~/.minikube/

# Reinstall VirtualBox

sudo apt-get install virtualbox

# Rebuild minikube VM

minikube start --driver=virtualbox

Solution in one line:

sudo killall VirtualBox; rm -rf ~/.config/VirtualBox/; rm -rf ~/.minikube/; sudo apt-get remove -y --purge virtualbox && sudo apt-get install  -y virtualbox && minikube start --driver=virtualbox

Convert PostGIS ST_LineString to ST_MultiPoint

Sometimes you need to convert a PostGIS line into a multipoint geometry. I mean, how else did you find this?

SELECT ST_Union(

    ARRAY(

        SELECT (ST_DumpPoints(linestringcolumn)).geom

        FROM atable

    )

);

Just kidding, use ST_Points(). Why ST_MultiPoint() does not do this is a mystery to my tiny brain. I would ask Dan Baston, but his answer would probably fracture my mental model of reality, and I don't have time to reconstruct it right now.

SELECT ST_Points(linestringcolumn) FROM atable;

Fix for npm ERR! command-line line 0: "unsupported option accept-new"

I encountered this on a server with a newer npm version and an older openSSH version. A few node dependencies were stored in private git repositories, and npm just simply wouldn't fetch them.

The fix was to downgrade npm to v6

    npm i -g npm@6

Unreal Engine 4 pointer cursor stuck on web browser widget

I know UE4's web browser widget is an "experimental" feature, but this is a pretty obnoxious bug that they have refused to fix for at least 4 years.

As far as I can tell, as soon as your cursor enters the "pointer" state while inside the browser widget, it will be stuck there indefinitely, even after you close the widget. The only solution I have found is to prevent it from ever reaching that state.

If you have full control over the webpage you are displaying, you can modify your CSS to use some other cursor type for all your elements. I did not.

My solution? Set the cursor for every element to 'default' using javascript.

var nodes = document.querySelectorAll('*');

for(var i=0; i<nodes.length; i++) {

    nodes[i].setAttribute('style', 'cursor:default !important'); 

}

When? All of the time.

It does seem excessive to do every element every tick, and it probably is, but when you have no control over a webpage, you never know when some new pesky element is going to be shoved into the DOM and screw you again. Why risk it?

One thing I did try was to set a software "Hand" cursor. I assume this translates to cursor:pointer, but who knows? That did nothing at all. Maybe the bug here is that there is no "pointer" cursor type in UE4, so the web browser widget has no idea how to turn it off.

One thing I did NOT try was to set up invisible hardware cursor images. Perhaps there is a conflict in the transition from hardware to software cursors? I do not know, but I do know that I end up with both when I mouse out of an HTML button. Forever.

Reducing Wordpress, PHP, MySQL Memory Usage

BACKGROUND

I hate WordPress, but I also love WordPress. It is still the most user-friendly open source blogging platform in existence. I have tried many alternatives, I've even written my own, but the feature set has always felt lacking.

My main problem with WordPress used to be the codebase. It was a mess, but the devs have managed to clean it up quite a bit. The current version seems quite tidy as far as PHP projects go, and hacking on it isn't nearly as frustrating as it had been in earlier versions.

So now my big beef is with performance. PHP and MySQL are slow, gobble up tons of resources even when sitting idle, and WordPress' extensive feature set dumps a ton of bloat on top.

At the end of the day, I just want users to load my dumb blog as fast as possible and have WordPress leave as small of a footprint as possible so I can save my server resources for more interesting projects.

HOW TO REDUCE MYSQL MEMORY USAGE

The first thing I notice when checking my system memory is MySQL. Why are you eating half of my memory, MySQL? You're not doing anything. Just give it back.

MySQL loves to cache. It increases query speed tremendously to store your entire database in memory, but that's not my goal. Once I'm done writing my post, I won't need MySQL anymore, because I will be generating a static version of my blog for public viewing. Nothing serves faster than static files, and I intend my blog to be read only for the general public.

My end goal now is to dial down MySQL's memory usage as far as possible while still being able to use the Wordpress admin panel without noticeable lag.

So open up your MySQL config (mine was at /etc/mysql/mysql.conf.d/mysqld.cnf) and start tweaking some settings.

The biggest single improvement that I was able to make was to turn off MySQL's Performance Schema by adding this line:

performance_schema = 0

I also found a huge list of variables here and then proceeded to completely ignore the "minimum" values:

key_buffer_size = 8
max_allowed_packet = 1K
max_connections = 10
query_cache_limit = 0
query_cache_size = 0
expire_logs_days = 1
max_binlog_size = 1K
thread_stack = 1
thread_cache_size = 0
host_cache_size = 0
innodb_buffer_pool_size = 0
innodb_log_buffer_size = 0
innodb_ft_cache_size = 0
innodb_ft_total_cache_size = 0
sort_buffer_size = 0
read_buffer_size = 0
read_rnd_buffer_size = 0
max_heap_table_size = 0
tmp_table_size = 0
bulk_insert_buffer_size = 0
join_buffer_size = 0
net_buffer_length = 0
innodb_sort_buffer_size = 0
binlog_cache_size = 0
binlog_stmt_cache_size = 0

Now just save and restart MySQL.

I'm amazed that WordPress can still function at all with these settings, but honestly I don't even notice the difference when clicking around the admin panel. But what I did notice what that I had slashed MySQL's memory usage by 90%.

HOW TO REDUCE PHP MEMORY USAGE

Just like MySQL, I will only need PHP to write posts, not to serve them. Again, I will be minimizing PHP's memory usage by modifying configuration files.

PHP's biggest issue is that, by default, it spawns multiple processes to handle incoming requests, and then refuses to kill them once the requests have been handled. These processes, even when sitting idle, take up way too much of my precious memory.

So most of the work can be done in the pool config. For my system, that file is at /etc/php/7.0/fpm/pool.d/www.conf

pm = ondemand
pm.process_idle_timeout = 1s

This configuration tells PHP to not create any listener processes until a request is actually made. If a process does nothing for more than 1 second, it will be killed. Humanely. (PETA plz no bully.)

So PHP can now eat all the memory it wants when I'm actually using it (almost never) and will give it back immediately after. If you are worried about peak memory usage, you can set 

pm.max_children = 1

Other than than you can't do much. Limiting a PHP thread's max memory size will result in errors. You are essentially at the mercy of WordPress' dev team.

HOW TO REDUCE WORDPRESS MEMORY USAGE

All of this is has been a lead up to the main objective, building WordPress into a static website. Static websites are faster, more stable, and more secure. They also require very little memory to serve.

If you don't mind manually generating your public-facing WordPress site, I would recommend installing the Simply Static WordPress Plugin, and just use that.

If you're lazy, I'll share how I was able to trigger a static site generation every time a post is published or updated.

The first step is to create a shell script to clone your WordPress site into a public directory. I called mine static.sh

# Download your dynamic wordpress site into /tmp

wget --trust-server-names --no-clobber --mirror --convert-links --adjust-extension -e robots=off -P /tmp http://privateblog.com

# Search for any mention of your private domain and replace it with your public domain
find /tmp/privateblog.com -type f -print0 | xargs -0 sed -i 's/privateblog\.com/publicblog\.com/g'

# Manually copy your uploads directory to make sure you have all the images sizes that wget ignored

cp -r /var/www/privateblog.com/wp-content/uploads/* /tmp/privateblog.com/wp-content/uploads/

# Move the result into your public www directory

rm -rf /var/www/publicblog/*

mv /tmp/privateblog.com/* /var/www/publicblog/

Now we have a bash script that needs to run whenever we update our blog. Luckily WordPress makes this easy by shipping with XML-RPC functionality.

Since XML-RPC uses HTTP as a transport protocol, we don't need any extra technology to detect a change; we can just use PHP. Of course, if you wanted to analyze the XML payload, you would need to set up an actual XML-RPC server, but that's not necessary here.

So now we just need a PHP script added to our WordPress directory which calls static.sh. I called mine rpc.php

<?php passthru('./static.sh'); ?>

Nice. The last piece of this is to add the URL of rpc.php to our WordPress Update Services. In the dashboard, go to Settings > Writing and find Update Services down at the bottom. Add your URL to the box, and hit Save Change.

That should do the trick. If you run into problems, double check your file permissions and ownership. Keep in mind that the www-data user needs to be able to execute both scripts and have write access to your public directory.