Uploading large files over public wifi

No Comments »

Anyone who has worked out of a coffee shop and has had to upload large files to a remote system has probably experienced the initial burst speed of the upload followed by a disheartening throttling down of bandwidth to a crawl.

As I write a lot of software that uses the JVM I produce some rather large JAR files which need to get copied to a remote Puppet host, and I experience this bandwidth throttling often – both on public wifi and when I use my cellular data plan. Fortunately there’s an easily solution if you’re working on a Unix platform – split a file and have multiple concurrent uploads where each upload can take advantage of the initial bandwidth burst.

The below script isn’t perfect but it gets the job done. I’ve limited the concurrent uploads to four and new batches of uploads are only started once the current batch has completed. I based this script off an article about running multiple processes in bash.

#!/bin/bash

FILE=$1
HOSTNAME=$2
USER=cfeduke
SPLIT_SIZE=3145728 # bytes, 3 MiB (1024*1024*3)
SCP_MAX_COUNT=4    # no. of maximum scps at once

cp $FILE /tmp
cd /tmp
FILE=$(basename $FILE)
split -b$SPLIT_SIZE $FILE file-part-

scp_count=0
for part in file-part-*; do
  scp -B "$part" $HOSTNAME:~ &
  scp_count=$(($scp_count+1))
  if [ "$scp_count" -gt $SCP_MAX_COUNT ]; then
    wait
    scp_count=0
  fi
done

if [ "$scp_count" -gt 0 ]; then
  wait
fi

ssh -o "BatchMode yes" -t $HOSTNAME "cat /home/$USER/file-part-* >> /home/$USER/$FILE && rm /home/$USER/file-part-*"
rm file-part-*

Akka and require for constructor argument validation

No Comments »

Problem: you have an Akka Actor with constructor arguments and you want to use require to perform argument verification (such as not null or any boolean validation). Then you want to test these assertions. When you run the test you can see the ActorSystem generating the exceptions and terminating with an ActorInitializationException complete with stack trace on your console but you can’t seem to capture the exception for verification to pass your unit test.

The solution is probably in the docs somewhere but I couldn’t seem to find it. Fortunately its pretty easy. Here I have an actor:

class StatsActor(val address: InetSocketAddress) extends Actor {
    require(address != null)
    ⋮

And here’s how to test it (in Specs2/ScalaTest WordSpec format):

⋮
    "initialized with a null address" should {
      "throw an exception" in new Environment {
        val probe = TestProbe()
        probe watch system.actorOf(StatsActor.props(null))
        probe expectMsgClass classOf[Terminated]
      }
    }

Solution from StackOverflow. I really like the EventListener approach but I haven’t given it a shot yet as the TestProbe met my needs with the least amount of code.


ARGF and ARGV at the same time in a Ruby CLI program

No Comments »

If you’ve tried to use ARGF in Ruby to accept piped input you might be in for a surprise when you attempt to add additional command-line arguments, such as:

$ cat something.log | your-ruby-program ~/outputfile.txt "some phrase"

Trying to read the ARGV arguments after using ARGF will cause your program to fail as it first attempts to open a file located at ~/outputfile.txt and read its contents – or even worse, succeed and not perform what you expect if such a file exists. Obviously not the desired goal.

Instead ARGV.shift can be used prior to invoking ARGF to peel those arguments off the argument stack:

#!/usr/bin/env ruby

output_path = ARGV.shift
search_phrase = ARGV.shift

if ARGV.length > 2
    limit = ARGV.shift
end

# now ARGF can be accessed without Ruby trying to automatically
# read files named after the supplied arguments
data = ARGF.read

Found this out after a bit of digging on StackOverflow and its in the Ruby docs too.


dyld: Symbol not found: _JRSUIGetKey (on OSX Mavericks)

No Comments »

Problem: dyld: Symbol not found: _JRSUIGetKey while trying to execute the GUI tools for Building a Modern Computer from First Principles (book).
Issue: You’ve installed a JDK 1.7 from Oracle because you do some sort of JVM language development and Apple only supports Java 1.6. Installing a 1.7 JDK removes the 1.6 Apple version, or when you upgraded to Mavericks you never bothered to install the Apple JDK 1.6.
Solution: Run a Linux VM with a GUI with a JDK 1.6 or 1.7. It works. Trying to get Apple-approved JDK 1.6 alongside a modern 1.7 JDK on OSX Mavericks is a nightmare with sanity draining effects.


Incompatible object argument for function call (in the JVM)

No Comments »

Exception message “Incompatible object argument for function call” with a nasty long stack trace, and it seems so generic that you may never find the answer.

What this almost always means to me, and what I almost always forget, is a library (dependency) version mismatch or more commonly that I’ve shoved a development version of a JAR into my applications lib/ directory to perform diagnostics and then subsequently updated the application – typically through a mechanism like an RPM installed by yum – where the development version of the JAR is still present in the classpath. Obviously this becomes a problem with wildcard classpath convenience, but easy to diagnose and fix, as long as it can be remembered. Or optionally perform some sort of cleanup in the %%postun in RPM for the lib/ directory, which is what I think I’ll impose.