Splunk & my Twitter archive

Recently I've been having a great time playing with Splunk. Splunk is a big data platform that allows you to search practically any machine data and present it in ways that will give you insight into what you have. It has practical applications for application management, IT operations, security, compliance, big data as well as web and business analytics.

I downloaded the free trial version, installed it locally and played with some personal data sources including phone bills, bank statements, my personal twitter archive as well as some weather data I downloaded from the Bureau of Meteorology.

Below are a few of the interesting charts that came out of my Twitter archive (@dan_cake), along with the basic search query used to extract and present the data in this way. Click to view a full-sized version.

 

Tweets by month

sourcetype=twitter_csv

Tweets peaked in July 2010 when I sent on average almost 4 tweets per day. The first drop in usage is probably due to the birth of my first child and then the subsequent months where there was hardly any usage is due to just being too busy at work and at home.

Tweets per hour of day

sourcetype=twitter_csv | stats count BY date_hour | chart sum(count) By date_hour

Most tweets were sent between 9am-5pm but there is an dip around lunchtime and an interesting smaller increase in usage between 9pm-11pm. What really surprised me about this was the volume of tweets sent between 1am and 5am. Drilling down into the data is seems that some of these are due to issues with the timezone of the device I was on.

Tweets sent by Twitter client

sourcetype=twitter_csv | rex field=client "<*>(?<client>.*)</a>" | eval client=lower(client) | top client

Also I was surprised by this. I know I have been searching for the perfect client but had forgotten just how many I have been through!

The search query involved stripping some HTML tags from some of the client values with regex as well as matching on lowercase to get around inconsistencies with the same client having different capitalisation.

Speed up your KeyRepeat rate OSX

defaults write NSGlobalDomain KeyRepeat -int 0

You'll need to log out and log in again to see the change

Slow localhost on OSX Lion

It seems with the Lion operating system update the addition of support for IPv6 addresses really slows down any localhost resolutions. As usual, a great answer on StackOverflow supplied the solution, capturing the IPv6 lookup as well as the standard IPv4. Thanks to guns for the solution.

Basically you need to add a value of ::1 for every localhost value:

127.0.0.1 mydomain.local
::1 mydomain.local

With regards to overriding domains in the hosts file, I have found that in some circumstances, Lion queries the IPv6 address for a domain if it senses that a domain is unreachable over the IPv4 network.

I discovered this when I noticed some ads that I had never seen before on Snow Leopard because I had redirected the ad domains to 127.0.0.1. I fired up wireshark and noticed AAAA (IPv6 DNS records) queries following the IPv4 A queries (IPv4). The ad servers indeed have IPv6 addesses and were able to serve me their content.

The solution to this is have a

::1 mydomain.com
entry for every

127.0.0.1 mydomain.com
entry in your hosts file.

Interestingly, if you happen to have a local webserver running on 127.0.0.1:80 and your browser receives a response from the webserver (error or otherwise), no AAAA query is issued, as it seems to be satisfied that a TCP connection was at least possible.

Link

Adding WordPress plugins

Recently I had some issues trying to add WordPress plugins through the dashboard. It seemed at first like a permissions issue but apparently with the latest version of WordPress you need to add a line of code to your wp-config.php file to enable this

define('FS_METHOD', 'direct');

This forces the file system method and enables you to add plugins again.

Change of content

Well it has been over a year since my last post, and in that time a great deal has changed. I moved countries, changed jobs, had a baby boy and bought a house!

Also during the last year my work focus has broadened away from being a Flash and ActionScript specialist. As Digital Director of a local agency I now manage all web projects covering a broad range of technology. This means that the posts I've been inspired to write are not really relevant to the existing content of this blog: "ActionScript Scraps". I have decided to continue blogging here, but will change the tone of the blog with new posts.

From now on I hope to be blogging on a wider variety of topics including all web technologies. Going along with this change I feel it is time for a design refresh which will be happening over the next few weeks.

Flash “Allow” webcam button not working – Firefox

I recently came across this problem with a small AR project I was working on. Sometimes the webcam would work, but other times the SWF would seem to freeze when the Webcam Flash Player Settings window would come up. I could still "tab" around the settings window but I couldn't select the "Allow" button to enable my webcam. This was pretty frustrating until I came across this helpful post on stackoverflow:
http://stackoverflow.com/questions/3003724/cant-click-allow-button-in-flash-on-firefox

It seems that it is a bug within the Flash Player Settings that occurs when the window width isn't a whole pixel, and the CSS for the window is set to margin auto. With a bit of javascript trickery you can make sure that you won't have this problem again.

Site Launch: How Much Carbon…?

This project was another fantastic visualisation for David McCandless of informationisbeautiful.
http://www.informationisbeautiful.net/

It is the second visualisation we have worked on, and it was even more fun than the first.
The data is coming in live (as a CSV) from a Google Docs Spreadsheet http://bit.ly/tonsofcarbon.
The application is built in Flash using ActionScript 3.0 and the PureMVC framework, with SWFAddress and a small amount of XML.

You can read more about this project on David's great blog here:
http://www.informationisbeautiful.net/2011/how-much-carbon/

link

Flash Builder 4 / Subversive / OSX / SVN

Recently I had to install the Subversive plug-in in Flash Builder 4, but was unable to due to consistent errors during installation.

FlashBuilder -> Help -> Install new Software
http://download.eclipse.org/technology/subversive/0.7/update-site/

I'm not sure if this is a bug in FlashBuilder 4, but this link just was just giving me errors during installation. If you are also having this problem, you can try another location that worked for me:

http://download.eclipse.org/releases/galileo/

thanks to http://www.flex-tutorial.fr/2010/03/30/flash-builder-4-installation-de-subversive-et-connecteurs-svn/ for the tip

UK Postcode Mapper Firefox extension is now open-source

I have now made the UK Postcode Mapper Firefox extension open source, after a user suggestion on this blog. I initially decided to create this extension as a way to learn more about Firefox and javascript. Since the first build I have updated it twice, but don't have time to maintain it anymore, so making it open-source makes perfect sense.
The source can be access on github at this location:
http://github.com/danjp/ukpostcodemapper

The license info is in the README file.
If you would like to get involved in publishing / updating this project at mozilla.org, please email me and I will add you as a new Author.

Waiting for Adobe Flash Player to connect to debugger

Recently a FlashBuilder project I was working on just stopped compiling using the debugger. The program would just hang with the build progress stuck on 57%. I tried many things to fix this, including uninstalling and reinstalling the debug versions of the Flash player without any luck. Eventually I found that the player that was launching was not the Debug player but the Release build. I discovered that once I changed the file preferences for SWF files it would debug again.
To do this on a Mac, select "Open with...", choose "Other...", select the Debug player and check the "Always Open With" checkbox.
There is another method document on StackOverflow that I haven't tried.
Mac OSX / FlashBuilder 4