Cleaning, moving, updating, adding :D

master
Taeyeon Mori 9 years ago
parent 079729918f
commit 5dddf9409d
  1. 3
      .gitignore
  2. 63
      README.md
  3. 61
      bin/argshell
  4. 26
      bin/aur.sh
  5. 3
      bin/force-exec
  6. 6
      bin/pulse-g930
  7. 4
      bin/pulse-restart
  8. 54
      bin/pulse-start
  9. 29
      bin/pulse-tunnels
  10. 151
      bin/xconv
  11. 2
      dotfiles/makepkg.conf
  12. 3
      etc/aur.conf
  13. 11
      etc/aur.conf~
  14. 31
      install
  15. 61
      lib/libpulse-config.sh
  16. 28
      lib/libsh-utils.sh
  17. 64
      lib/libzsh-utils.zsh
  18. 757
      lib/python/advancedav.py
  19. 44
      zsh/lib.zsh
  20. 2
      zsh/prezto
  21. 40
      zsh/zprofile
  22. 7
      zsh/zshenv
  23. 1
      zsh/zshrc

3
.gitignore vendored

@ -1,4 +1,7 @@
/zsh/.*
user-info
*.local
__pycache__
*.pyc
*~

@ -1,31 +1,76 @@
Taeyeon's dotfiles (.files)
===========================
As of right now, only contains dotfiles for zsh, vim and git.
.files/install
--------------
A ZShell script to set up the dotfiles. It will also ask a few personal questions.
Run this on a new system to set it up
Run this on a new system to set it up.
.files/zsh
----------
The ZSH part makes use of the Prezto framework (http://github.com/sorin-ionescu/prezto)
Configuration happens in the z\* files in .files/zsh:
- zshenv : executed by all zsh instances
- zprofile : executed by any top-level shell
- zshrc : executed by interactive shells (loads prezto)
- preztorc : contains prezto-specific configuration
- zlogin : executed by login shells
- zlogout : executed when a login shell exits
.files/bin
----------
Contains utility applications I'd hate to miss
Notable ones are:
- xconv : A simple, profile-based, batch-enabled "frontend" to ffmpeg.
- aconvert : Pre-advancedav version of xconv. Has profiles for extracting audio.
- ffpsp\* : Scripts to run HandBrakeCLI to convert videos for use with Sony's PSP.
- pulse-\* : Tools to be used with my pulse configuration in .files/config/pulse
- aur.sh : More powerfull version of the popular aur.sh script.
- schedshut : Shutdown when a specific task/process finishes.
- mountpart : Mount a partition in a whole-disk imagefile.
- argshell : call the same application repeatedly, with a common set of arguments.
.files/etc
----------
Contains configuration for those utilities
Currently contains:
- aur.conf : Configuration for aur.sh
- user-info : The user information entered at install time, in shell-readable form
.files/lib
----------
Contains support libraries
Currently, that entails:
- libsh-utils.sh : A collection of useful shell functions
- libzsh-utils.zsh : More utility shell functions, but using zsh-specific features
- libpulse-config.sh: Functions for working with the pulseaudio configs in .files/config/pulse
- libssh-agent.sh : Functions for working with the ssh-agent
As well as some python modules:
- advancedav.py : A very overengineered way to construct complex ffmpeg commandlines
.files/git
----------
Contains the git configuration (.files/git/config)
\*/user-info
-----------
Contains user information collected at install time.
currently only .files/user-info and .files/git/user-info
NOTE that changes made though git config WILL NOT BE RESPECTED,
because it writes to ~/.gitconfig, which is a proxy that includes .files/git/config .
Also note that git uses its own version of user-info (.files/git/user-info)
.files/dotfiles
---------------
General dotfiles repository. Everything that doesn't need to be special-cased goes here.
.files/config
-------------
XDG configuration directory.
Items will be symlinked to ~/.config (NOT IMPLEMENTED)
$HOME
-----
All dotfiles are symlinked into the home directory or have a proxy file generated.
@ -36,3 +81,7 @@ Known proxies: .gitconfig, .zshenv
--------
Files ending on .local are ignored by git.
Currently, valid .local files are:
- zsh/zprofile.local
- zsh/zshrc.local

@ -0,0 +1,61 @@
#!/bin/zsh -i
# .files|Argshell
# (c) 2015 Taeyeon Mori
# Synopsis
function synopsis() {
print "Synopsis: $1 [-h] [-e] [-l|-ll] <command...>"
print ""
print "Run a command repeatedly, with different arguments."
print ""
print "Arguments:"
print " -h Show this help message"
print " -l Treat additional arguments as literal"
print " -ll Treat the whole command as literal"
print ""
print "[.files|argshell] Ver 1.0 (c) 2015 Taeyeon Mori"
}
# Parse options
EVAL_LINE=true
EVAL_CMD=true
while true; do
if [ "$1" = "-l" ]; then
EVAL_LINE=false
elif [ "$1" = "-ll" ]; then
EVAL_LINE=false
EVAL_CMD=false
elif [ "$1" = "-h" ]; then
synopsis "$0"
exit 0
else
break
fi
shift
done
if [ $# -lt 1 ]; then
synopsis "$0"
exit 1
fi
# The prompt
PROMPT="$* => "
prompt() {
echo -n "$2"
read $1
}
# Do the work
while prompt args "$PROMPT"; do
if $EVAL_LINE; then
eval "$@" $args
elif $EVAL_CMD; then
eval "$@" `python -c "import shlex;print(' '.join(\"'%s'\" % s.replace(\"'\",\"'\''\") for s in shlex.split(r'''$args''')))"`
else
"$@" $args
fi
done

@ -1,7 +1,8 @@
#!/bin/zsh
# vim: ft=sh:ts=2:sw=2:et
source "$DOTFILES/zsh/lib.zsh"
# Load libraries and configuraion
source "$DOTFILES/lib/libzsh-utils.zsh"
source "$DOTFILES/etc/aur.conf"
function throw {
@ -9,21 +10,33 @@ function throw {
exit $1
}
# Figure out build directory
tmpbuild=$TMPDIR/aur.sh.$$
build="${BUILDDIR:-$tmpbuild}"
test -d "$build" || mkdir -p "$build" || exit 1
# Parse commandline: anything prefixed with - is a makepkg option
packages=(${@##-*})
makepkg_flags=(${@##[^\-]*})
if echo "${makepkg_flags[*]}" | grep -q "\\-X"; then
warn "[AUR] Building was disabled (-X)"
DL_ONLY=true
else
DL_ONLY=false
fi
# Print some info
msg "[AUR] AURDIR=$AURDIR; PKGDEST=$PKGDEST"
test "$build" = "$PWD" || \
msg "[AUR] Working in $build."
msg "[AUR] Building packages: $packages"
# Process packages
for p in "${packages[@]}"; do
cd "$AURDIR"
# First, download the PKGBUILD from AUR, to $AURDEST
cd "$AURDEST"
msg "[AUR] $p: Getting PKGBUILD"
{
test -d $p && \
@ -34,8 +47,13 @@ for p in "${packages[@]}"; do
{ curl https://aur.archlinux.org/packages/${p:0:2}/$p/$p.tar.gz | tar xz } || \
throw 2 "[AUR] $p: Couldn't download package"
cd $p
if $DL_ONLY; then continue; fi
# Copy it to the build directory $build and change there
cp -r "$p" "$build"
cd "$build/$p"
# Run makepkg
msg "[AUR] $p: Building..."
makepkg "${makepkg_flags[@]}" || \
throw 1 "[AUR] $p: Makepkg failed!"
@ -45,6 +63,8 @@ done
msg "[AUR] All Done!"
# Remove the builddir if we previously created it.
cd "$AURDEST"
test "$build" = "$tmpbuild" && \
warn "[AUR] Removing temporary directory $tmpbuild" && \
rm -rf "$tmpbuild"

@ -0,0 +1,3 @@
#!/bin/sh
/lib/ld-linux-x86-64.so.2 "$1"

@ -0,0 +1,6 @@
#!/bin/sh
echo "Setting up pulseaudio tunnels for the G930 Headset"
echo -n "G930 Sink Tunnel: "
pactl load-module module-tunnel-sink-new server=10.66.64.4 sink=alsa_output.usb-Logitech_Logitech_G930_Headset-00-Headset.analog-stereo sink_name=tunnel.arch.g930 sink_properties=\"device.description=\'G930 Headset\'\"
echo -n "G930 Source Tunnel: "
pactl load-module module-tunnel-source-new server=10.66.64.4 source=alsa_input.usb-Logitech_Logitech_G930_Headset-00-Headset.analog-mono source_name=tunnel.arch.g930 source_properties=\"device.description=\'G930 Headset\'\"

@ -0,0 +1,4 @@
#!/bin/sh
pulseaudio -k
exec pulse-start

@ -0,0 +1,54 @@
#!/bin/bash
# PulseAudio startup script
# (c) 2014-2015 Taeyeon Mori
# Do what the fuck you want with it. (No warranty, etc)
# This is an advanced version of start-pulseaudio-x11 with per-host
# configuration support.
# Please note that to make use of it, pulseaudio must not be set up
# to start on demand.
# You can achieve this by setting "autospawn" to "no" in client.conf.
# Copy it from /etc/pulse/client.conf if it doesn't exist.
# You'll also have to set up some kind of automatic start for this script
# if you want pulseaudio to be available automatically (you just disabled autospawn!)
set -e
source "${DOTFILES-`dirname "$0"`/..}/lib/libpulse-config.sh"
# Dry run
test "$1" = "-dry" && function exec { echo $@; }
# == Check for machine-specific config file
pa_find_host_config
test -n "$PA_HOST_CONFIG" && STARTUP_FILE="$PA_HOST_CONFIG"
# == Additional Modules ==
declare -a LOAD_MODULES
# Register with X11.
# Adapted from start-pulseaudio-x11 script.
if [ x"$DISPLAY" != x ]; then
color 34 echo "Registering with X11 Display $DISPLAY."
push LOAD_MODULES "module-x11-publish display=$DISPLAY"
#push LOAD_MODULES "module-x11-cork-request display=$DISPLAY"
if [ x"$SESSION_MANAGER" != x ]; then
push LOAD_MODULES "module-x11-xsmp display=$DISPLAY session_manager=$SESSION_MANAGER"
fi
fi
# == Build Arguments ==
declare -a ARGS=(--start)
if test -n "$STARTUP_FILE"; then
push ARGS -nF "$STARTUP_FILE"
fi
for (( i=0; i<${#LOAD_MODULES[@]}; i++ )); do
push ARGS -L "${LOAD_MODULES[i]}"
done
# == Run pulseaudio ==
exec "$PA_EXECUTABLE" "${ARGS[@]}" "$@"

@ -0,0 +1,29 @@
#!/bin/bash
# PulseAudio tunnel script
# (c) 2014-2015 Taeyeon Mori
# Do what the fuck you want with it. (No warranty, etc)
# This script will scrape the PA config file for tunnel modules
# and load them via pactl.
# This is useful to recover from network outages and similar things.
set -e
source "${DOTFILES-$HOME/.files}/lib/libpulse-config.sh"
# Dry run
test "$1" = "-dry" && function pactl() { echo $@; }
# Work on the tunnels
pa_find_config_lines "load-module module-tunnel"
for tunnel in "${PA_CONFIG_LINES[@]}"; do
pa_parse_tunnel_line "$tunnel"
echo -en "\033[32mSetting up tunnel to $TUN_SERVER/$TUN_DEVICE: $TUN_NAME "
test -n "$TUN_DESCRIPTION" && echo -n "$TUN_DESCRIPTION "
echo -en "\033[31m>> "
pactl $tunnel || echo "Couldn't setup tunnel: $tunnel"
echo -en "\033[0m"
done

@ -0,0 +1,151 @@
#!/usr/bin/python3
"""
xconv ffpmeg wrapper based on AdvancedAV
-----------------------------------------------------------
AdvancedAV helps with constructing FFmpeg commandline arguments.
It can automatically parse input files with the help of FFmpeg's ffprobe tool (WiP)
and allows programatically mapping streams to output files and setting metadata on them.
-----------------------------------------------------------
Copyright (c) 2015 Taeyeon Mori
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see <http://www.gnu.org/licenses/>.
"""
from advancedav import SimpleAV
from argparse import ArgumentParser
from abc import ABCMeta, abstractmethod
from itertools import chain
from os.path import isdir, join as build_path, basename, splitext, exists
# == Profile Decorators ==
profiles = {}
def profile(f):
profiles[f.__name__] = f
return f
def output(container="matroska", ext="mkv"):
def output(f):
f.container = container
f.ext = ext
return f
return output
# == Profile definitions ==
@profile
@output(container="matroska", ext="mkv")
def laptop(task):
# enable experimental aac codec
task.options["strict"] = "-2"
# add first video stream
for s in task.iter_video_streams():
os = task.map_stream(s)
os.options["codec"] = "libx264"
os.options["vf"] = "scale='if(gt(a,16/10),1280,-1)':'if(gt(a,16/10),-1,800)'" # scale to 1280x800, keeping the aspect ratio
os.options["tune"] = "fastdecode", "animation"
os.options["profile"] = "main"
os.options["preset"] = "fast"
break
# Add all audio streams (reencode to aac if necessary)
for s in task.iter_audio_streams():
os = task.map_stream(s)
if s.codec != "aac":
os.options["codec"] = "aac"
# add all subtitle and attachment streams
for s in chain(task.iter_subtitle_streams(), task.iter_attachment_streams()):
os = task.map_stream(s)
# go
return True
# == Support code ==
def parse_args(argv):
parser = ArgumentParser(prog=argv[0])
files = parser.add_argument_group("Files")
files.add_argument("inputs", nargs="+", help="The input file(s)")
files.add_argument("output", help="The output filename or directory")
parser.add_argument("-p", "--profile", choices=profiles.keys(), required=True, help="Specify the profile. See the source code.")
parser.add_argument("-m", "--merge", help="Merge streams from all inputs instead of mapping each input to an output", action="store_true")
parser.add_argument("-u", "--update", help="Only work on files that don't already exist", action="store_true")
progs = parser.add_argument_group("Programs")
progs.add_argument("--ffmpeg", default="ffmpeg", help="Path to the ffmpeg executable")
progs.add_argument("--ffprobe", default="ffprobe", help="Path to the ffprobe executable")
return parser.parse_args(argv[1:])
def make_outfile(args, infile):
if args.genout:
if hasattr(args.profile, "ext"):
return build_path(args.output, ".".join((splitext(basename(infile))[0], args.profile.ext)))
else:
return build_path(args.output, basename(infile))
else:
return args.output
def main(argv):
import logging
logging.basicConfig(level=logging.DEBUG)
args = parse_args(argv)
args.profile = profiles[args.profile]
args.genout = isdir(args.output)
if not args.genout:
if len(args.inputs) > 1:
print("Output path '%s' is not a directory." % args.output)
return -1
aav = SimpleAV(ffmpeg=args.ffmpeg, ffprobe=args.ffprobe)
tasks = []
if args.merge:
task = aav.create_job(make_outfile(args, args.inputs[0]))
for input in args.inputs:
task.add_input(input)
tasks.append(task)
else:
for input in args.inputs:
out = make_outfile(args, input)
if args.update and exists(out):
continue
task = aav.create_job(out, args.profile.container if hasattr(args.profile, "container") else None)
task.add_input(input)
tasks.append(task)
for task in tasks:
if not args.profile(task):
print("Failed to apply profile for '%s'" % basename(task.name))
return 1
for task in tasks:
task.commit()
return 0
if __name__ == "__main__":
import sys
sys.exit(main(sys.argv))

@ -1,7 +1,7 @@
# vim: ft=sh
# Packager Info
source "$DOTFILES/user-info"
source "$DOTFILES/etc/user-info"
PACKAGER="$REALNAME <$EMAIL>"
# Optimizations

@ -2,10 +2,11 @@
# Folders
AURDIR="$HOME/aur"
AURDEST="$AURDIR/Pkgbuilds"
PKGDEST="$AURDIR/Packages"
SRCDEST="$AURDIR/Source"
test -d "$AURDIR" || mkdir -p "$AURDIR" || exit 1
test -d "$AURDEST" || mkdir -p "$AURDEST" || exit 1
test -d "$PKGDEST" || mkdir -p "$PKGDEST" || exit 1
test -d "$SRCDEST" || mkdir -p "$SRCDEST" || exit 1

@ -1,11 +0,0 @@
# vim: ft=sh
# Folders
AURDIR="$HOME/aur"
PKGDEST="$AURDIR/.pkg"
SRCDEST="$AURDIR/.sauce"
test -d "$AURDIR" || mkdir -p "$AURDIR" || exit 1
test -d "$PKGDEST" || mkdir -p "$PKGDEST" || exit 1
test -d "$SRCDEST" || mkdir -p "$SRCDEST" || exit 1

@ -4,9 +4,10 @@
# Paths & Utils
DOT="$(realpath "$(dirname "$0")")"
cd "$DOT"
setopt EXTENDED_GLOB
source "$DOT/zsh/lib.zsh"
source "$DOT/lib/libzsh-utils.zsh"
# Parse commandline
OVERWRITE=false
@ -54,7 +55,7 @@ function relink() {
[[ ! -e "$1" ]] && err "No such file or directory: '$1'" && return 1
[[ -L "$2" ]] && rm -f "$2"
[[ -e "$2" ]] && ! $OVERWRITE && err "File already exists and isn't a symbolic link: '$2'" && return 1
color 31 ln -s "$1" "$2"
color 31 ln -s "$(relpath "$1" "$(dirname "$2")")" "$2"
}
# Update git modules
@ -64,7 +65,7 @@ git submodule update --init --recursive
# Ask questions about user
$ASK && msg "Asking questions..." || err "Not asking any questions:"
[[ -e "$DOT/user-info" ]] && source "$DOT/user-info"
[[ -e "$DOT/etc/user-info" ]] && source "$DOT/etc/user-info"
[[ -z "$REALNAME" ]] && REALNAME=`grep "^$USER:[^:]*:$UID:" /etc/passwd | cut -d: -f5 | cut -d, -f1`
@ -76,7 +77,7 @@ else
color 36 echo " default EMAIL=\"$EMAIL\""
fi
generate "$DOT/user-info" <<EOF
generate "$DOT/etc/user-info" <<EOF
REALNAME="$REALNAME"
EMAIL="$EMAIL"
EOF
@ -97,8 +98,8 @@ GIT="$DOT/git"
generate "$HOME/.gitconfig" <<EOF
[include]
path = $GIT/config
path = $GIT/user-info
path = $(relpath "$GIT" "$HOME")/config
path = $(relpath "$GIT" "$HOME")/user-info
EOF
generate "$GIT/user-info" <<EOF
@ -112,21 +113,23 @@ msg "Setting up ZSH configuration..."
ZSH="$DOT/zsh"
generate "$HOME/.zshenv" <<EOF
export DOTFILES="$DOT"
export DOTFILES="\$HOME/$(relpath "$DOT" "$HOME")"
ZDOTDIR="\$DOTFILES/zsh"
source "\$ZDOTDIR/zshenv"
EOF
for file in zprofile zshrc zpreztorc zlogin zlogout; do
for file in zshrc zpreztorc zlogin zlogout; do
relink "$ZSH/$file" "$ZSH/.$file"
done
relink "$ZSH/prezto" "$ZSH/.zprezto"
# Set login shell
msg "Setting login shell to '/bin/zsh'..."
echo $SHELL | grep -q zsh || chsh -s /bin/zsh
# Warn user
color 33 echo "You need to relog to apply shell configuration."
# Check login shell
if echo $SHELL | grep -q zsh; then
msg "Login shell is already set to zsh, you're good to go!"
else
msg "Setting login shell to '/bin/zsh'..."
chsh -s /bin/zsh
warn echo "You need to relog to apply shell configuration."
fi

@ -0,0 +1,61 @@
#!/bin/bash
# PulseAudio startup script
# (c) 2014-2015 Taeyeon Mori
# Do what the fuck you want with it. (No warranty, etc)
source "${DOTFILES-$(dirname "${BASH_SOURCE-$0}")/..}/lib/libsh-utils.sh" #"
# Paths
PA_CONFIG_DIR="${XDG_CONFIG_DIR-$HOME/.config}/pulse"
PA_HOSTS_DIR="$PA_CONFIG_DIR/machines.d"
# PulseAudio
PA_EXECUTABLE="/usr/bin/pulseaudio"
# == Look for a host-specific config file. ==
# It should be located in $MACHINES_DIR and named "<hostname>.pa", with <hostname> all lower-case.
# If none is found, PA will be started using the default files ($PULSE_DIR/default.pa, /etc/pulse/default.pa)
pa_find_host_config() {
test -n "$PA_HOST_CONFIG" && return
declare -g PA_HOST_CONFIG_NAME="${HOSTNAME,,}.pa"
declare -g PA_HOST_CONFIG_FILE="$PA_HOSTS_DIR/$PA_HOST_CONFIG_NAME"
if test -e "$PA_HOST_CONFIG_FILE"; then
color 33 echo "Using machine configuration: $PA_HOST_CONFIG_NAME"
declare -g PA_HOST_CONFIG="$PA_HOST_CONFIG_FILE"
fi
}
pa_find_config() {
test -n "$PA_CONFIG" && return
pa_find_host_config
if test -n "$PA_HOST_CONFIG"; then
declare -g PA_CONFIG="$PA_HOST_CONFIG"
elif test -e "$PA_CONFIG_DIR/default.pa"; then
color 33 echo "Using default configuration: default.pa"
declare -g PA_CONFIG="$PA_CONFIG_DIR/default.pa"
elif test -e "/etc/pulse/default.pa"; then
color 33 echo "Using global default configuration: /etc/pulse/default.pa"
declare -g PA_CONFIG="/etc/pulse/default.pa"
fi
}
# Parse tunnel load-module line
pa_parse_tunnel_line() {
declare -g TUN_NAME=`echo $1 | grep -oP '(sink|source)_name=[^ ]+' | cut -f2 -d=`
declare -g TUN_SERVER=`echo $1 | grep -oP 'server=[^ ]+' | cut -f2 -d=`
declare -g TUN_DEVICE=`echo $1 | grep -oP '(sink|source)=[^ ]+' | cut -f2 -d=`
declare -g TUN_DESCRIPTION=`echo $1 | grep -oP "device.description='[^']+'" | cut -f2 -d=`
}
pa_find_config_lines() {
if test -n "$2"; then
local cfg="$2"
else
pa_find_config
local cfg="$PA_CONFIG"
fi
local IFS=$'\n'
declare -ga PA_CONFIG_LINES=(`grep "$1" "$cfg"`)
}

@ -0,0 +1,28 @@
# Shell Utitlities library
# Should work in both bash and zsh
# (c) 2014-2015 Taeyeon Mori
# I <3 predicates :)
# [predicate] Mute command
quiet() {
"$@" >/dev/null 2>&1
return $?
}
# [predicate] Colorize output
color() {
local COLOR=$1 && shift
echo -en "\e[${COLOR}m"
"$@"
echo -en "\e[0m"
return $?
}
# Append to array
push() {
local arr="$1"; shift
for val in "$@"; do
eval "$arr[\${#$arr[@]}]=\"\$val\""
done
}

@ -0,0 +1,64 @@
# ZSH Utitlities library
# Zsh-only extensions to libsh-utils.sh
# (c) 2014-2015 Taeyeon Mori
# I <3 predicates :)
source "${0%/*}/libsh-utils.sh"
alias msg="color 34 echo"
alias warn="color 33 echo"
alias err="color 31 echo"
# Ask a question
function ask {
local RESULT
echo -en "\e[35m$1"
[[ -n "${(P)2}" ]] && echo -n " [${(P)2}]"
echo -n ": "
color 36 read $3 RESULT
[[ -n "$RESULT" ]] && eval $2=\""$RESULT"\"
}
# Get random choice
function random_choice {
if quiet which shuf; then
shuf -n1 -e "$@"
else
local NUMBER=${RANDOM%$#+1}
echo ${(P)NUMBER}
fi
}
# Print the a relative path from the second directory to the first,
# defaulting the second directory to $PWD if none is specified.
# SOURCE: http://www.zsh.org/mla/users/2002/msg00267.html
function relpath {
[[ $1 != /* ]] && print $1 && return
local dir=${2:-$PWD}
[[ $1 == $dir ]] && print . && return
local -a cur abs
cur=(${(ps:/:)dir}) # Split 'current' directory into cur
abs=(${(ps:/:)1}) # Split target directory into abs
local min
((min = $#cur < $#abs ? $#cur : $#abs))
local i=1
while ((i <= $min)) && [[ $abs[1] == $cur[$i] ]]
do
abs[1]=() # Strip common prefix from target directory
((i=i+1))
done
# Figure out how many parents to get to common root
local relpath=
while ((i <= $#cur))
do
relpath=../$relpath
((i=i+1))
done
relpath=$relpath${(j:/:)abs}
print ${relpath%/}
}

@ -0,0 +1,757 @@
"""
AdvancedAV FFmpeg commandline generator v2.0 [Library Edition]
-----------------------------------------------------------
AdvancedAV helps with constructing FFmpeg commandline arguments.
It can automatically parse input files with the help of FFmpeg's ffprobe tool (WiP)
and allows programatically mapping streams to output files and setting metadata on them.
-----------------------------------------------------------
Copyright 2014-2015 Taeyeon Mori
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see <http://www.gnu.org/licenses/>.
"""
import os
import re
import subprocess
import collections
import itertools
from abc import ABCMeta, abstractmethod
from collections.abc import Iterable, Mapping, Sequence, Iterator, MutableMapping
__all__ = "AdvancedAVError", "AdvancedAV", "SimpleAV"
# Constants
DEFAULT_CONTAINER = "matroska"
S_AUDIO = "a"
S_VIDEO = "v"
S_SUBTITLE = "s"
S_ATTACHMENT = "t"
S_DATA = "d"
S_UNKNOWN = "u"
def stream_type(type_: str) -> str:
""" Convert the ff-/avprobe type output to the notation used on the ffmpeg/avconv commandline """
lookup = {
"Audio": S_AUDIO,
"Video": S_VIDEO,
"Subtitle": S_SUBTITLE,
"Attachment": S_ATTACHMENT,
"Data": S_DATA
}
return lookup.get(type_, S_UNKNOWN)
class AdvancedAVError(Exception):
pass
class Stream:
"""
Abstract stream base class
One continuous stream of data muxed into a container format
"""
__slots__ = "file", "type", "index", "pertype_index", "codec", "profile"
def __init__(self, file: "File", type_: str, index: int=None, pertype_index: int=None,
codec: str=None, profile: str=None):
self.file = file
self.type = type_
self.index = index
self.pertype_index = pertype_index
self.codec = codec
self.profile = profile
def update_indices(self, index: int, pertype_index: int=None):
""" Update the Stream indices """
self.index = index
if pertype_index is not None:
self.pertype_index = pertype_index
@property
def stream_spec(self):
""" The StreamSpecification in the form of "<type>:<#stream_of_type>" or "<#stream>" """
if self.pertype_index is not None:
return "{}:{}".format(self.type, self.pertype_index)
else:
return str(self.index)
def __str__(self):
return "<%s %s#%i: %s %s (%s)>" % (type(self).__name__, self.file.name, self.index,
self.type, self.codec, self.profile)
class InputStream(Stream):
"""
Holds information about an input stream
"""
__slots__ = "language"
def __init__(self, file: "InputFile", type_: str, index: int, language: str, codec: str, profile: str):
super().__init__(file, type_, index, codec=codec, profile=profile)
self.file = file
self.language = language
def update_indices(self, index: int, pertype_index: int=None):
""" InputStreams should not have their indices changed. """
if index != self.index:
raise ValueError("Cannot update indices on InputStreams! (This might mean there are bogus ids in the input")
# pertype_index gets updated by File._add_stream() so we don't throw up if it gets updated
class OutputStream(Stream):
"""
Holds information about a mapped output stream
"""
__slots__ = "source", "metadata", "options"
# TODO: support other parameters like frame resolution
# Override polymorphic types
#file = None
""" :type: OutputFile """
def __init__(self, file: "OutputFile", source: InputStream, stream_id: int, stream_pertype_id: int=None,
codec: str=None, options: Mapping=None, metadata: MutableMapping=None):
super().__init__(file, source.type, stream_id, stream_pertype_id, codec)
self.source = source
self.options = options if options is not None else {}
self.metadata = metadata if metadata is not None else {}
# -- Manage Stream Metadata
def set_meta(self, key: str, value: str):
""" Set a metadata key """
self.metadata[key] = value
def get_meta(self, key: str) -> str:
""" Retrieve a metadata key """
return self.metadata[key]
class File:
"""
ABC for Input- and Output-Files
"""
__slots__ = "name", "options", "_streams", "_streams_by_type"
def __init__(self, name: str, options: dict=None):
self.name = name
self.options = options if options is not None else {}
""" :type: dict[str, str] """
self._streams = []
""" :type: list[Stream] """
self._streams_by_type = collections.defaultdict(list)
""" :type: dict[str, list[Stream]] """
def _add_stream(self, stream: Stream):
""" Add a stream """
stream.update_indices(len(self._streams), len(self._streams_by_type[stream.type]))
self._streams.append(stream)
self._streams_by_type[stream.type].append(stream)
@property
def streams(self) -> Sequence:
""" The streams contained in this file
:rtype: Sequence[Stream]
"""
return self._streams
@property
def video_streams(self) -> Sequence:
""" All video streams
:rtype: Sequence[Stream]
"""
return self._streams_by_type[S_VIDEO]
@property
def audio_streams(self) -> Sequence:
""" All audio streams
:rtype: Sequence[Stream]
"""
return self._streams_by_type[S_AUDIO]
@property
def subtitle_streams(self) -> Sequence:
""" All subtitle streams
:rtype: Sequence[Stream]
"""
return self._streams_by_type[S_SUBTITLE]
@property
def attachment_streams(self) -> Sequence:
""" All attachment streams (i.e. Fonts)
:rtype: Sequence[Stream]
"""
return self._streams_by_type[S_ATTACHMENT]
@property
def data_streams(self) -> Sequence:
""" All data streams
:rtype: Sequence[Stream]
"""
return self._streams_by_type[S_DATA]
@property
def filename(self) -> str:
""" Alias for .name """
return self.name
def __str__(self):
return "<%s %s>" % (type(self).__name__, self.name)
class InputFile(File):
"""
Holds information about an input file
"""
__slots__ = "pp", "_streams_initialized"
def __init__(self, pp: "AdvancedAV", filename: str, options: Mapping=None):
super().__init__(filename, dict(options.items()) if options else None)
self.pp = pp
self._streams_initialized = False
# -- Probe streams
_reg_probe_streams = re.compile(
r"Stream #0:(?P<id>\d+)(?:\((?P<lang>\w+)\))?: (?P<type>\w+): (?P<codec>\w+)"
r"(?: \((?P<profile>[^\)]+)\))?(?P<extra>.+)?"
)
def _initialize_streams(self, probe: str=None) -> Iterator:
""" Parse the ffprobe output
The locale of the probe output in \param probe should be C!
:rtype: Iterator[InputStream]
"""
if probe is None:
probe = self.pp.call_probe(("-i", self.name))
for match in self._reg_probe_streams.finditer(probe):
self._add_stream(InputStream(self,
stream_type(match.group("type")),
int(match.group("id")),
match.group("lang"),
match.group("codec"),
match.group("profile")))
self._streams_initialized = True
@property
def streams(self) -> Sequence:
""" Collect the available streams
:rtype: Sequence[InputStream]
"""
if not self._streams_initialized:
self._initialize_streams()
return self._streams
@property
def video_streams(self) -> Sequence:
""" All video streams
:rtype: Sequence[InputStream]
"""
if not self._streams_initialized:
self._initialize_streams()
return self._streams_by_type[S_VIDEO]
@property
def audio_streams(self) -> Sequence:
""" All audio streams
:rtype: Sequence[InputStream]
"""
if not self._streams_initialized:
self._initialize_streams()
return self._streams_by_type[S_AUDIO]
@property
def subtitle_streams(self) -> Sequence:
""" All subtitle streams
:rtype: Sequence[InputStream]
"""
if not self._streams_initialized:
self._initialize_streams()
return self._streams_by_type[S_SUBTITLE]
class OutputFile(File):
"""
Holds information about an output file
"""
__slots__ = "task", "container", "metadata", "_mapped_sources"
def __init__(self, task: "Task", name: str, container=DEFAULT_CONTAINER, options: Mapping=None):
# Set default options
_options = {"c": "copy"}
if options is not None:
_options.update(options)
# Contstuct
super().__init__(name, _options)
self.task = task
self.container = container
self.metadata = {}
""" :type: dict[str, str] """
self._mapped_sources = set()
""" :type: set[InputStream] """
# -- Map Streams
def map_stream_(self, stream: InputStream, codec: str=None, options: Mapping=None) -> OutputStream:
""" map_stream() minus add_input_file
map_stream() needs to ensure that the file the stream originates from is registered as input to this Task.
However, when called repeatedly on streams of the same file, that is superflous.
"""
out = OutputStream(self, stream, -1, -1, codec, options)
self._add_stream(out)
self._mapped_sources.add(stream)
self.task.pp.to_debug("Mapping Stream %s => %s (%i)",
self.task.qualified_input_stream_spec(stream),
out.stream_spec,
self.task.outputs.index(self))
return out
def map_stream(self, stream: InputStream, codec: str=None, options: Mapping=None) -> OutputStream:
""" Map an input stream to the output
Note that this will add multiple copies of an input stream to the output when called multiple times
on the same input stream. Check with is_stream_mapped() beforehand if the stream might already be mapped.
"""
self.task.add_input(stream.file)
return self.map_stream_(stream, codec, options)
def is_stream_mapped(self, stream: InputStream) -> bool:
""" Test if an input stream is already mapped """
return stream in self._mapped_sources
def get_mapped_stream(self, stream: InputStream) -> OutputStream:
""" Get the output stream this input stream is mapped to """
for out in self._streams:
if out.source == stream:
return out
# -- Map multiple Streams
def map_all_streams(self, file: "str | InputFile", return_existing: bool=False) -> Sequence:
""" Map all streams in \param file
Note that this will only map streams that are not already mapped.
:rtype: Sequence[OutputStream]
"""
out_streams = []
for stream in self.task.add_input(file).streams:
if stream in self._mapped_sources:
if return_existing:
out_streams.append(self.get_mapped_stream(stream))
else:
out_streams.append(self.map_stream_(stream))
return out_streams
def merge_all_files(self, files: Iterable, return_existing: bool=False) -> Sequence:
""" Map all streams from multiple files
Like map_all_streams(), this will only map streams that are not already mapped.
:type files: Iterable[str | InputFile]
:rtype: Sequence[OutputStream]
"""
out_streams = []
for file in files:
for stream in self.task.add_input(file).streams:
if stream in self._mapped_sources:
if return_existing:
out_streams.append(self.get_mapped_stream(stream))
else:
out_streams.append(self.map_stream_(stream))
return out_streams
# -- Sort Streams
def reorder_streams(self):
""" Sort the mapped streams by type """
self._streams.clear()
for stream in itertools.chain(self.video_streams,
self.audio_streams,
self.subtitle_streams):
stream.update_indices(len(self._streams))
self._streams.append(stream)
# -- Manage Global Metadata
def set_meta(self, key: str, value: str):
self.metadata[key] = value
def get_meta(self, key: str) -> str:
return self.metadata[key]
class Task:
"""
Holds information about an AV-processing Task.
A Task is a collection of Input- and Output-Files and related options.
While OutputFiles are bound to one task at a time, InputFiles can be reused across Tasks.
"""
def __init__(self, pp: "AdvancedAV"):
self.pp = pp
self.inputs = []
""" :type: list[InputFile] """
self.inputs_by_name = {}
""" :type: dict[str, InputFile] """
self.outputs = []
""" :type: list[OutputFile] """
# -- Manage Inputs
def add_input(self, file: "str | InputFile") -> InputFile:
""" Register an input file
When \param file is already registered as input file to this Task, do nothing.
:param file: Can be either the filename of an input file or an InputFile object.
The latter will be created if the former is passed.
"""
if not isinstance(file, InputFile):
if file in self.inputs_by_name:
return self.inputs_by_name[file]
file = InputFile(self.pp, file)
if file not in self.inputs:
self.pp.to_debug("Adding input file #%i: %s", len(self.inputs), file.name)
self.inputs.append(file)
self.inputs_by_name[file.name] = file
return file
def qualified_input_stream_spec(self, stream: InputStream) -> str:
""" Construct the qualified input stream spec (combination of input file number and stream spec)
None will be returned if stream's file isn't registered as an input to this Task
"""
file_index = self.inputs.index(stream.file)
if file_index >= 0:
return "{}:{}".format(file_index, stream.stream_spec)
# -- Manage Outputs
def add_output(self, filename: str, container: str=DEFAULT_CONTAINER, options: Mapping=None) -> OutputFile:
""" Add an output file
NOTE: Contrary to add_input this will NOT take an OutputFile instance and return it.
"""
for outfile in self.outputs:
if outfile.name == filename:
raise AdvancedAVError("Output File '%s' already added." % filename)
else:
outfile = OutputFile(self, filename, container, options)
self.pp.to_debug("New output file #%i: %s", len(self.outputs), filename)
self.outputs.append(outfile)
return outfile
# -- Manage Streams
def iter_video_streams(self) -> Iterator:
for input_ in self.inputs:
yield from input_.video_streams
def iter_audio_streams(self) -> Iterator:
for input_ in self.inputs:
yield from input_.audio_streams
def iter_subtitle_streams(self) -> Iterator:
for input_ in self.inputs:
yield from input_.subtitle_streams
def iter_attachment_streams(self) -> Iterator:
for input_ in self.inputs:
yield from input_.attachment_streams
def iter_data_streams(self) -> Iterator:
for input_ in self.inputs:
yield from input_.data_streams
# -- FFmpeg
@staticmethod
def argv_options(options: Mapping, qualifier: str=None) -> Iterator:
""" Yield arbitrary options
:type options: Mapping[str, str]
:rtype: Iterator[str]
"""
if qualifier is None:
opt_fmt = "-%s"
else:
opt_fmt = "-%%s:%s" % qualifier
for option, value in options.items():
yield opt_fmt % option
if isinstance(value, (tuple, list)):
yield value[0]
for x in value[1:]:
yield opt_fmt % option
yield x
elif value is not None:
yield value
@staticmethod
def argv_metadata(metadata: Mapping, qualifier: str=None) -> Iterator:
""" Yield arbitrary metadata
:type metadata: Mapping[str, str]
:rtype: Iterator[str]
"""
if qualifier is None:
opt = "-metadata"
else:
opt = "-metadata:%s" % qualifier
for meta in metadata.items():
yield opt
yield "%s=%s" % meta
def generate_args(self) -> Iterator:
""" Generate the ffmpeg commandline for this task
:rtype: Iterator[str]
"""
# Inputs
for input_ in self.inputs:
# Input options
yield from self.argv_options(input_.options)
# Add Input
yield "-i"
filename = input_.name
if filename[0] == '-':
yield "./" + filename
else:
yield filename
# Outputs
for output in self.outputs:
# Global Metadata & Additional Options
yield from self.argv_metadata(output.metadata)
yield from self.argv_options(output.options)
# Map Streams, sorted by type
output.reorder_streams()
for stream in output.streams:
yield "-map"
yield self.qualified_input_stream_spec(stream.source)
if stream.codec is not None:
yield "-c:%s" % stream.stream_spec
yield stream.codec
yield from self.argv_metadata(stream.metadata, stream.stream_spec)
yield from self.argv_options(stream.options, stream.stream_spec)
# Container
if output.container is not None:
yield "-f"
yield output.container
# Output Filename, prevent it from being interpreted as option
out_fn = output.name
yield out_fn if out_fn[0] != "-" else "./" + out_fn
def commit(self, additional_args: Iterable=()):
"""
Commit the changes.
additional_args is used to pass global arguments to ffmpeg. (like -y)
:type additional_args: Iterable[str]
:raises: AdvancedAVError when FFmpeg fails
"""
self.pp.call_conv(itertools.chain(additional_args, self.generate_args()))
class SimpleTask(Task):
"""
A simple task with only one output file
All members of the OutputFile can be accessed on the SimpleTask directly, as well as the usual Task methods.
Usage of add_output should be avoided however, because it would lead to confusion.
"""
def __init__(self, pp: "AdvancedAV", filename: str, container: str=DEFAULT_CONTAINER, options: Mapping=None):
super().__init__(pp)
self.output = self.add_output(filename, container, options)
def __getattr__(self, attr: str):
""" You can directly access the OutputFile from the SimpleTask instance """
return getattr(self.output, attr)
# Allow assignment to these OutputFile members
def _redir(attr, name):
def redir_get(self):
return getattr(getattr(self, attr), name)
def redir_set(self, value):
setattr(getattr(self, attr), name, value)
return property(redir_get, redir_set)
container = _redir("output", "container")
metadata = _redir("output", "metadata")
name = _redir("output", "name")
options = _redir("output", "options")
del _redir
class AdvancedAV(metaclass=ABCMeta):
# ---- Output ----
@abstractmethod
def to_screen(self, text: str, *fmt):
""" Log messages to the user """
pass
@abstractmethod
def to_debug(self, text: str, *fmt):
""" Process verbose messages """
pass
# ---- FFmpeg ----
@abstractmethod
def call_conv(self, args: Iterable):
"""
Call ffmpeg.
:param args: Iterable[str] The ffprobe arguments
It should throw an AdvancedAVError if the call fails
"""
pass
@abstractmethod
def call_probe(self, args: Iterable) -> str:
"""
Call ffprobe.
:param args: Iterable[str] The ffprobe arguments
:return: str the standard output
It should throw an AdvancedAVError if the call fails
NOTE: Make sure the locale is set to C so the regexes match
"""
pass
# ---- Create Tasks ----
def create_task(self) -> Task:
"""
Create a AdvancedAV Task.
"""
return Task(self)
def create_job(self, filename: str, container: str=DEFAULT_CONTAINER, options: Mapping=None) -> SimpleTask:
"""
Create a simple AdvandecAV task
:param filename: str The resulting filename
:param container: str The output container format
:param options: Additional Options for the output file
:return: SimpleTask An AdvancedAV Task
"""
return SimpleTask(self, filename, container, options)
# ---- Create InputFiles ----
def create_input(self, filename: str, options):
"""
Create a InputFile instance
:param filename: str The filename
:param optiona: Mapping Additional Options
:return: A InputFile instance
NOTE that Task.add_input is usually the preferred way to create inputs
"""
return InputFile(self, filename, options)
class SimpleAV(AdvancedAV):
"""
A simple Implementation of the AdvancedAV interface.
It uses the python logging module for messages and expects the ffmpeg/ffprobe executables as arguments
"""
def __init__(self, *, ffmpeg="ffmpeg", ffprobe="ffprobe", logger=None, ffmpeg_output=True):
if logger is None:
import logging
self.logger = logging.getLogger("advancedav.SimpleAV")
else:
self.logger = logger
self._ffmpeg = ffmpeg
self._ffprobe = ffprobe
self.ffmpeg_output = ffmpeg_output
self.logger.debug("SimpleAV initialized.")
def to_screen(self, text, *fmt):
self.logger.log(text % fmt)
def to_debug(self, text, *fmt):
self.logger.debug(text % fmt)
_posix_env = dict(os.environ)
_posix_env["LANG"] = _posix_env["LC_ALL"] = "C"
def call_conv(self, args: Iterable):
""" Actually call ffmpeg
:type args: Iterable[str]
"""
argv = tuple(itertools.chain((self._ffmpeg,), args))
self.to_debug("Running Command: %s", argv)
output = None if self.ffmpeg_output else subprocess.DEVNULL
subprocess.call(argv, stdout=output, stderr=output)
def call_probe(self, args: Iterable):
""" Call ffprobe (With LANG=LC_ALL=C)
:type args: Iterable[str]
"""
argv = tuple(itertools.chain((self._ffprobe,), args))
self.to_debug("Running Command: %s", argv)
proc = subprocess.Popen(argv, stdout=subprocess.PIPE, stderr=subprocess.PIPE, env=self._posix_env)
out, err = proc.communicate()
if proc.returncode != 0:
err = err.decode("utf-8", "replace")
msg = err.strip().split('\n')[-1]
raise AdvancedAVError(msg)
return err.decode("utf-8", "replace")

@ -1,44 +0,0 @@
#!/bin/zsh
# ZSH Utitlities library
# (c) 2014 MORI Taeyeon
# I <3 predicates :)
# [predicate] Mute command
function quiet {
"$@" >/dev/null 2>&1
return $?
}
# [predicate] Colorize output
function color {
local COLOR=$1 && shift
echo -en "\e[${COLOR}m"
"$@"
echo -en "\e[0m"
return $?
}
alias msg="color 34 echo"
alias warn="color 33 echo"
alias err="color 31 echo"
# Ask a question
function ask() {
local RESULT
echo -en "\e[35m$1"
[[ -n "${(P)2}" ]] && echo -n " [${(P)2}]"
echo -n ": "
color 36 read $3 RESULT
[[ -n "$RESULT" ]] && eval $2=\""$RESULT"\"
}
# Get random choice
function random_choice() {
if quiet which shuf; then
shuf -n1 -e "$@"
else
local NUMBER=${RANDOM%$#+1}
echo ${(P)NUMBER}
fi
}

@ -1 +1 @@
Subproject commit 10cf7018645fd9c09a0aa03c4ebac5ae6222c77e
Subproject commit ee0b02464ca29b551169d0a33a43aeef979d78ea

@ -1,8 +1,24 @@
#
# Executes commands at login pre-zshrc.
# Executes commands in every top-level shell
#
# Authors:
# Sorin Ionescu <sorin.ionescu@gmail.com>
# ATTENTION: This differs from usual zsh behaviour!
# Treat this file as a per-session zshenv instead.
#
# Why is this useful? Because it only gets executed once for any given tree
# of zsh-processes. You can use it to do environment computations that would
# be too expensive for zshenv.
#
# Then why also source it in non-interactive shells?
# Because it prevents desync between scripts and interactive prompts.
# Ever tried to debug a script and ended up with "But it works in the prompt!"?
#
# That is especially true with X11-Scenarios, since a display-manager usually
# doesn't source any profile. Which will lead to shells without a login shell in
# their parent process tree, which means the profile is basically useless.
#
# If you need loginshell-only commands, use zlogin instead. If you insist on
# zprofile, create a .zprofile (note the dot) file in this directory and zsh will
# treat it as always.
#
#
@ -18,7 +34,7 @@ fi
#
export EDITOR='vim'
export VISUAL='nano'
export VISUAL='vim'
export PAGER='less'
#
@ -33,9 +49,6 @@ fi
# Paths
#
# Ensure path arrays do not contain duplicates.
typeset -gU cdpath fpath mailpath path
# Set the the list of directories that cd searches.
# cdpath=(
# $cdpath
@ -50,6 +63,18 @@ path=(
$path
)
# Set the list of directories that Python searches for modules.
pythonpath=(
$DOTFILES/lib/python
"${(@s|:|)PYTHONPATH}"
)
# Ensure path arrays do not contain duplicates.
typeset -gU cdpath fpath mailpath path pythonpath
# zsh only maps the array and text versions for PATH
export PYTHONPATH="${(j|:|)pythonpath}"
#
# Less
#
@ -80,4 +105,3 @@ fi
# Local overrides
[[ -e "$DOTFILES/zsh/zprofile.local" ]] && source "$DOTFILES/zsh/zprofile.local"

@ -5,8 +5,7 @@
# Sorin Ionescu <sorin.ionescu@gmail.com>
#
# Ensure that a non-login, non-interactive shell has a defined environment.
if [[ "$SHLVL" -eq 1 && ! -o LOGIN && -s "${ZDOTDIR:-$HOME}/.zprofile" ]]; then
source "${ZDOTDIR:-$HOME}/.zprofile"
# Source the profile on every top-level shell
if [[ "$SHLVL" -eq 1 ]]; then
source "$DOTFILES/zsh/zprofile"
fi

@ -35,4 +35,3 @@ function gupf {
# Local overrides
[[ -e "$DOTFILES/zsh/zshrc.local" ]] && source "$DOTFILES/zsh/zshrc.local"

Loading…
Cancel
Save