July 17, 2013

Linux Users: Are You Über Leet? Show Us Your Hacks

by with 22 comments

Leet greetz to all the linux hackers out there. System Administrator Appreciation Day is next week (July 26). Because:

  1. SpiderOak has a large number of Linux users (inside and out)
  2. The Linux community has long been a do-it-yourself crowd of folks willing to solve their own problems…

We want to tap into your expertise. We know you’ve got skillz so join us for SysAdmin Day! Enter, and you might win 100GB/year.

So give us a shout, via a blog comment, on Facebook, or Twitter:

  • Are you uber leet? Show us your Hacks!
  • Give us your best script (Bash, Perl, Python, whatever!), commandline hack (–datadir ftw!) or anything that gives you bragging rights (ever figured out that prefs.dat is portable? Why? That’s gotta be a good story.)
  • What quirky, zany and inventive ways do you use SpiderOak on Linux?
  • Tips & tricks?
  • Do you have any Linux stories we should hear?

In a week, we will publish the TOP 10 LINUX HACKS for SpiderOak Users. If you make the list, you win 100GB for a year. (Last entry will be taken 11:59 CT on Wed. July 24.)

And heck – if you tell us a story that is that K-rad, we’ve got 50GB honorable mention prizes. Come at us, bro!

Thanks for being such a strong, loyal community. You be some badass h4ck3rz.

Comments
  1. It’s not super fancy, but a quick little script to batch process dvd/blu-ray rips and move them into my Plex movie library.

    —–

    cd /media/raid5/rips/ondeck/
    for f in *
    do
    HandBrakeCLI -Z “High Profile” -i “$f” -o “../processed/$f.mp4″
    rm -rf “$f”
    chown plex:plex “../processed/$f.mp4″
    mv -f “../processed/$f.mp4″ ../../plexLibrary/movies/
    date
    done

  2. I have a desktop in the office, at the house, and a laptop that floats around, all running Gentoo Linux; naturally I want access to them all at anytime and anyplace. So my problem was: how do I figure out my ip address, both on the LAN and to the external world, and then update all my computers to have that information? Keep in mind that at the office I have no control over the network, router, and so on.

    I suppose a traditional answer would be something like purchase a domain name with a dynamic update to the ip address (e.g., dyndns.com), point it to the home desktop, and then have reverse ssh tunnels running all the time. The downside is the cost of the domain name and making sure that the desktop and tunnels are always up and running. Plus, the big disadvantage is the network overhead. If I happen to have the laptop in the office, going from it to the office desktop would involve a round trip through the internet instead of just staying on the LAN.

    So I thought, “I could skip the domain name with an updated ~/.ssh/config, plus I could add entries for the LAN ip and have the best solution. But how am I going to keep them updated without an ssh connection? Updated… in sync… sync… SPIDEROAK!” So I whipped up something quick in Python that runs at login and periodically through cron, and set spideroak to sync my ssh config file on all machine (yes, it could probably be done better in bash, but multi-line sed replacements are hard!).

    I know, the script could use some tiding up (e.g., wireless interfaces have different names on different machines, so for the time being I wrote that in by hand)… but it works, damnit!

    -- ~/.ssh/config --
    # Snipping some global options
    Host                 zeal_palace
    Hostname             internet-ip-address
    
    Host                 zeal_palace-local
    Hostname             192.168.1.100
    
    Host                 kajar
    Hostname             internet-ip-address
    
    Host                 kajar-local
    Hostname             192.168.0.110
    
    # You get the picture...
    
    -- update-ssh.py -- 
    
    #! /bin/env python3
    
    import os
    import re
    import subprocess
    
    def execute(cmd, capture=False):
        """
        Execute a command line process.  Includes the option of capturing output,
        and checks for successful execution.
        """
        #print(cmd)
    
        with open(os.devnull, 'w') as void:
            if capture:
                sub = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=void)
            else:
                sub = subprocess.Popen(cmd, shell=True, stdout=void, stderr=void)
        status = sub.wait()
    
        # Exit if the command fails for any reason.
        if status != 0:
            sys.exit(1)
    
        if capture:
            text = sub.stdout.read().decode("utf-8")
            text = text.replace('\n', '')
            return text
        else:
            return None
    
    host = execute('hostname', True)
    addr = execute('curl --max-time="60" --silent ifconfig.me', True)
    
    with open('/home/strider1551/.ssh/config', 'r') as handle:
        haystack = handle.read()
        needle = 'Host\s+{0}\nHostname\s+[a-zA-Z0-9.]+'.format(host)
        replacement = 'Host                 {0}\nHostname             {1}'.format(host, addr)
        haystack = re.sub(needle, replacement, haystack)
    
        if host == "enhasa":
            addr = execute("ifconfig wlp8s0", True)
            addr = re.findall('inet\s[a-zA-Z0-9.]+', addr)[0]
            addr = addr[5:]
            needle = 'Host\s+{0}-local\nHostname\s+[a-zA-Z0-9.]+'.format(host)
            replacement = 'Host                 {0}-local\nHostname             {1}'.format(host, addr)
            haystack = re.sub(needle, replacement, haystack)
        if host == "kajar":
            addr = execute("ifconfig wlp11s0", True)
            addr = re.findall('inet\s[a-zA-Z0-9.]+', addr)[0]
            addr = addr[5:]
            needle = 'Host\s+{0}-local\nHostname\s+[a-zA-Z0-9.]+'.format(host)
            replacement = 'Host                 {0}-local\nHostname             {1}'.format(host, addr)
            haystack = re.sub(needle, replacement, haystack)
    
    with open('/home/strider1551/.ssh/config', 'w') as handle:
        handle.write(haystack)
    
    • That is a bit dangerous. Assuming you have a desktop environment setup (with the traditional trash bin) you should look into setting up rm-trash and setting alias rm=rm-trash instead. It simply moves the items to the trash bin instead of deleting them then. And emptying the trash bin once you’re sure you don’t need anything in there anymore is pretty simple and fast. Not going to lie, it’s saved me time on a few occasions when I accidentally rm’d the wrong file.

  3. I made the SpiderOak sync with my home bin dir and point the PATH to the bin.

    Since I have many self made Bash and Ruby codes, I know that if I write one and save it, all my servers will have latest copies of code and immediately accessible.

    I also sync git repository this way(and the normal commit method), I can restore via git or from the SpiderOak when I need to do a rollback, knowing that I still have copies of code on other machines not just on the clouds gives me peace of mind.

  4. I’m no Uber, but I have pet peaves I want taken care of. I don’t like dealing with updates, like at all, don’t want to deal with any maintanence notifications. Those make me feel like a slave to my machine. So on startup I have a script to do a robust and clean update on ubuntu and keep a good log of things incase anything happens. Recently I took off the automatic dist-upgrade because of a hardware issue with my lappy. But I’ll probably uncomment that again, maybe. I don’t only use this script everyday but literally everytime my computer is turned on. It’s very refreshing not to care about updates but know you’re completely updated. I also made a cron version for servers. (just took of the sleep and added to crontab)
    http://pastebin.com/RjCqBrmQ
    (The log file is written last entry first so you don’t have to scroll to see what the last action was)

    I have this other semi-uncompleted script that searches for rootkits, everytime my computer turns on. But I haven’t expanded it to any good kind of notification. I haven’t settled on a notification I’d be comfortable with. So I guess in that sense it’s kind of useless, but it’s good to have it so I can run it when I want and check the log files manually, periodically.
    http://pastebin.com/3Af5afWn

    These last two scripts some other spideroak users might like more.
    I like to use truecrypt but I don’t like using passwords for them, but keyfiles. I like lots of complicated keyfiles that can easily be hidden as anything, but I don’t like the work involved in making them. This keyfile generator originally used randomsound for lots of entropy but I stopped using that for haveged because of compatibility issues. I know software entropy generation isn’t the best but it’s good enough for my uses.
    http://pastebin.com/DBWQ5q6a
    (I had to research how to get a mid range number to make file sizes more randomly and found a super complicated function in a bash book. Many years later on commandlinefu.com I found an easy one liner. But I worked so dam hard on that freaking thing I decided to leave it in all it’s mess)

    There’s been so much debate, i.e legitimate scientific debate, on password length and type and how long and what’s exactly the safest. So I took the infamous inspiriration from https://xkcd.com/936, commandlinefu oneliners, and my own practice on what makes a strong – yet memorable – password. I’m very indecisive and run this thing like 10 times before I find one I like and even then I don’t go with exactly what comes out of it. Only used when I want to remember a good password but otherwise I just use a regular random one.
    http://pastebin.com/jzU1Grrw
    (I think some advanced password cracker could come up with a plausible amount of password hashes knowing your password attributes, according to some reasearch by Arstechnica, but I’m not sure that’s been proven in this case (the script way). Anyone know?)

    Sorry I have more than one script but these ones I use ALL the time. They are synced on every debian machine I have, with SpiderOak ofcourse ;)

  5. I’ve recently discovered the rsync backup and tasker apps for Android. This allows me to sync my files from my Ubuntu Server to my Android Tablet. Of course I won’t need it once SpiderOak offer automated full file synchronisation on Android devices…. ;-)

  6. My easy hack – list of commands you use most often, simple and powerful:

    history | awk ‘{a[$'`echo "1 2 $HISTTIMEFORMAT" | wc -w`']++}END{for(i in a){print a[i] “t” i}}’ | sort -rn | head

  7. # SpiderOak completion
    # This should be placed in the /etc/bash_completion.d/ directory

    _SpiderOak()
    {
    local cur prev opts
    COMPREPLY=()
    cur=”${COMP_WORDS[COMP_CWORD]}”
    prev=”${COMP_WORDS[COMP_CWORD-1]}”
    opts=”–help –version –verbose –redirect –device –output –enable-schedule –disable-schedule –scan –scan-only –build –scan-and-build-only –sync –backup –restore –batchmode –headless –merge –purge –userinfo –user-info –shelved-x –print-shelved-x –space –tree –fulllist –tree-changelog –journal-changelog –force –selection –print-selection –reset-selection –exclude-file –exclude-dir –include-dir –rebuild-reference-database –billing –destroy-shelved-x –repair –purge-historical-versions –purge-deleted-items –vacuum –list-orphan-external-files –convert-to-hybrid-db –generate-previews –bootstrap –setup –list-shares –create-share –delete-share”

    case “${prev}” in
    –output | –exclude-dir | –include-dir | –purge-deleted-items | –journal-changelog)
    COMPREPLY=( $(compgen -d ${cur}) )
    return
    ;;
    –redirect | –exclude-file | –bootstrap | –setup | –backup | –restore | –purge)
    COMPREPLY=( $(compgen -f ${cur}) )
    return
    ;;
    *)
    ;;
    esac

    COMPREPLY=( $(compgen -W “${opts}” — ${cur}) )
    }
    complete -F _SpiderOak SpiderOak

  8. Many people know that you can add the name of the Git branch you are currently working in to the end of your command prompt but did you know that you can also have the command prompt notify you if you have any untracked of modified files?

    First I place the following in ~/.bashrc so that .bashrc can read my script.

    # Include my very own git parse type function :-)
    source ~/.GitParseStatus.sh

    Then I update the PS1 variable (the command prompt text) to output the result of the script

    export PS1=”\n${debian_chroot:+($debian_chroot)}${RED}\u@\H ${YELLOW}\A ${BLUE}\w \$(gitParseStatus)${NORMAL}\n$ ”

    It’s the \$(gitParseStatus) bit

    and then of course I have the script which I save as ~/.GitParseStatus.sh (don’t forget to make the script executable with chmod +x ~/.GitParseStatus.sh)

    ——————- Start of ~/.GitParseStatus——————
    #!/bin/bash
    #function get_out() {
    # log $1
    # return
    #}
    #
    #function log() {
    # echo ‘* ‘$1
    #}
    #
    #
    #if [ $1 = "--debug" ]; then
    # DEBUG=true
    # echo
    # echo “===========”
    # echo “Debug is on”
    # echo “===========”
    # echo
    #fi
    #
    #=========================
    # Set Variables
    #=========================

    # What do you want to show when everything is up to date
    # Your options are…
    TICK=$’\u2714′
    OK=’OK’
    CUSTOM_OK=”

    # Now set the all ok message
    ALL_OK=$TICK

    # What do you want to display when there git needs updating in some way?
    # I have used “M” for files have been modified
    UPDATE_TEXT=’ M’

    # What do you want to display when there are untracked files in the repo?
    # I have used “U” but you can use whatever you want
    UNTRACKED_TEXT=” U”

    # How do you want to display the repo branch
    # I have used square brackets
    BRANCH_PRE_TEXT=”["
    BRANCH_POST_TEXT="]”

    # What colour do you want the git section of you command line to be when
    # everything is ok?
    # I have it as green
    # Set some colours #
    # Some examples are…
    GIT_RESET=”17″
    GIT_NORMAL=”33[0m"
    GIT_RED_1="33[31;1m"
    GIT_YELLOW_1="33[33;1m"
    GIT_WHITE="33[37;1m"
    GIT_RED="33[0;31m"
    GIT_YELLOW="33[01;33m"
    GIT_GREEN="33[0;32m"
    GIT_BLUE="33[01;034m"
    BRANCH_COLOUR_ALL_OK=$GIT_GREEN
    BRANCH_COLOUR_NOT_OK=$GIT_GREEN

    TEXT_CODES_COLOUR_ALL_OK=$GIT_GREEN
    TEXT_CODES_COLOUR_NOT_OK=$GIT_RED

    function gitParseStatus {

    # If we fail to get a branch name then simply return empty handed :-(
    #IN=$(git symbolic-ref HEAD 2> /dev/null) || echo "You are not in a Git repo"
    IN=$(git symbolic-ref HEAD 2> /dev/null) || return
    #echo $IN

    BRANCH_NAME=`echo $IN | sed 's/refs\/heads\///'`
    #log 'Branch Name="'$BRANCH_NAME'"'

    # get a list of files to run through
    FILES=$(git status -z 2> /dev/null)
    #log 'Git Status="'"$FILES"'"'

    #echo

    if [[ -z "$FILES" ]]; then

    #log “All the files in your Git repo are up to date”
    UPDATES=`echo -e ${TEXT_CODES_COLOUR_ALL_OK}`${ALL_OK}

    BRANCH=`echo -e ${BRANCH_COLOUR_ALL_OK}`”$BRANCH_PRE_TEXT”"$BRANCH_NAME”"$BRANCH_POST_TEXT”

    OUTPUT=”$BRANCH”

    else
    #log “You have descrepancies in your Git Repo”
    BRANCH=`echo -e ${BRANCH_COLOUR_NOT_OK}`”$BRANCH_PRE_TEXT”"$BRANCH_NAME”"$BRANCH_POST_TEXT”

    #log ‘Clearing the $UPDATES variable’
    UPDATES=”"

    #echo ‘About to start testing on “‘${FILES}’”‘
    #echo
    #log ‘Testing for untracked files (ones beginning with ??)’
    UNTRACKED=$(echo ${FILES} | grep $’\x000?{2}’)
    #echo ‘UNTRACKED=”‘$UNTRACKED’”‘

    if [ -n "$UNTRACKED" ] ; then

    #log “Untracked files were found”
    UPDATES+=”$UNTRACKED_TEXT”

    else
    UNTRACKED=$(echo ${FILES} | grep ^?{2})
    #echo ‘UNTRACKED=”‘$UNTRACKED’”‘

    if [ -n "$UNTRACKED" ] ; then

    #log “Untracked files were found”
    UPDATES+=”$UNTRACKED_TEXT”
    fi

    #log “No Untracked files were found”
    fi

    #echo
    #log “Testing for modified files”

    modified=$(echo ${files} | grep $’\x000[rmdc ]{2}’)
    #echo ‘modified=”‘$modified’”‘
    if [ -n "$modified" ] ; then

    #log “modified files were found”
    updates+=”$update_text”

    else
    modified=$(echo ${files} | grep ‘^[rmdc ]{2}’)
    #echo ‘modified=”‘$modified’”‘
    if [ -n "$modified" ] ; then

    #log “modified files were found”
    updates+=”$update_text”
    fi

    #log “No Modified files were found”
    fi

    UPDATE_OUTPUT=`echo -e ${TEXT_CODES_COLOUR_NOT_OK}`$UPDATES

    OUTPUT=$BRANCH$UPDATE_OUTPUT

    fi

    echo $OUTPUT

    }
    ————— End of ~/.GitParseStatus.sh————–

    Enjoy :-)
    John (Grandad)

  9. #! /bin/sh

    # for extra anonymity on hotspots, change the MAC address of your radio before associating.
    # requires the wicd-cli client and the easymac virtual device MAC generation script from:
    # http://www.easyvmx.com/software/easymac.sh

    # note – already created /etc/wpa_suplicant/wpa_supplicant.conf
    # by running $ wpa_passphrase

    # note – the gnome network-manager service will conflict with wicd, so ditch it.

    clear

    # disable built-in broadcom radio
    modprobe -r wl

    # disable ethernet
    ifconfig eth0 down

    count=0
    success=1
    while [ $success -eq 1 ];do
    count=$(($count+1))
    echo “Attempt #”$count

    # bring the interface down
    echo “Bringing interface down…”
    ifconfig wlan1 down

    # generate a new MAC
    echo “Generating new MAC…”
    newMAC=$(/opt/scripts/easymac.sh -g -m)

    # try to bring it up
    echo “Trying to change MAC…”
    ifconfig wlan1 hw ether $newMAC

    # check the success
    success=$?
    echo “Checking success…”
    done
    echo “MAC changed to “$newMAC “on try #”$count

    # bring the interface up
    echo “Bringing the interface up…”
    ifconfig wlan1 up

    echo “Tearing down any previous connection…”
    wicd-cli -y -x

    echo “Scanning for networks…”
    wicd-cli -y -S

    echo “Available networks:”
    wicd-cli -y -l

    echo
    echo -n “Connect to which network number? ”
    read netnum

    echo “Connecting…”
    wicd-cli -y -c -n $netnum

  10. Stop spideroak eating all my 3G data when I’m on the road:

    In crontab:

    /home/joel/bin/stop_spideroak_when_on_3g.sh

    $ cat /home/joel/Dropbox/bin/stop_spideroak_when_on_3g.sh
    if (nm-tool | grep -A 4 Bigpond | grep connected 2>1 >/dev/null) ||
    (nm-tool | grep -A 4 Ultimate | grep connected 2>1 >/dev/null); then
    SPIDEROAK_PID=`ps aux | grep ‘SpiderOak$’ | sed ‘s/\s\+/ /g’ | cut -d’ ‘ -f2`
    kill -QUIT $SPIDEROAK_PID
    fi

  11. Not really a hack, but I had a script in a cronjob that would tar and gzip all of my logs in ~/.SpiderOak so that they wouldn’t rotate out but then I found out that there’s actually these env variables:

    SPIDEROAK_LOG_ROTATE_COUNT # 10
    SPIDEROAK_LOG_LEVEL # debug
    SPIDEROAK_LOG_ROTATE_SIZE # 10*1024*1024 (10M)

    So now I’ve got logging until I run out of space.