Skip to main content

Raspberry Pi Download Machine Part 1

Yet Another Raspberry Pi Project

Kodi builds, weird fucking robots and classic gaming always seems to be the theme of RPi projects. But a stat I would like to know is how many are just sat in a drawer somewhere. Anyway I am part of that number, until now. I manage to find a 4TB drive and managed to blag a caddy, so with this triangle of 'crap hanging around' I thought I'd build a downloader.


Open Directories

So I first had the idea from the sub-reddit open directories, here storage devices are opened up to the internet without a morsel of authentication. Google comes along and indexes them and you can go and find them. Decent ones find themselves on this sub of which I assume stays up until the user realises that there broadband is getting absolutely raped.

So I wanted to be able to get a URL from a server and just throw it in a WebUI and get it to auto download and ping me when all is finished. 


First Steps

So to start off I downloaded Raspbian Jessie Lite from 2017-07-05, then formatted my 4TB drive. Not gonna go through that process as enough cunts on the internet have done already. I have given my Pi a static address by binding the Mac Address in the router. Now you don't have to do this but makes life a bit easier when you need to refer to it.

Got my it all stuff plugged in and power and ready to rock and roll.

Oh wait, my first bit of fuckery already. I need to connect to the Pi and I am doing that using the terminal, so I need to SSH (remote connect for the command line). Seems that this distro hasn't started the SSH server by default. Thanks cunts.

(So at this point I angrily find an old TV and keyboard to plug into my PI, of course I then find out I run out of plug sockets so then spend half an hour digging out an extension lead. From doing that I then find another Raspberry Pi).

Log in with the username 'pi' and the password 'raspberry'. Then the easiest way is to run the config tool.

pi@raspberrypi:~ $ sudo raspi-config

Go to Interfacing Options then SSH. Select 'enable'


Now on you computer open Terminal or Putty and SSH with the details from before. On Terminal it's 

dom$ ssh pi@192.168.28.3

The enter the password 'raspberry'

Ok now is time to rock and roll. First 2 commands to run is to update and upgrade all packages. Think of this like going into the app store and clicking 'Update All'. (I am gonna omit the return text here). When prompted hit 'y' then Enter to install. (about 10mins)

pi@raspberrypi:~ $ sudo apt-get update
pi@raspberrypi:~ $ sudo apt-get upgrade


Mount your Drive

So, normally plug in a drive and job done, not here my little cherubs. In Unix world when you plug in a drive it has to exist inside a folder/directory (ok I am one of those pricks that swap between saying folder and directory so just think of them as the same). To attach the drive to the system in a specified directory is called 'mounting'. The following command will show you what is mounted where:


pi@raspberrypi:~ $ df -h

Filesystem      Size  Used Avail Use% Mounted on

/dev/root       7.3G  910M  6.1G  13% /
devtmpfs        458M     0  458M   0% /dev
tmpfs           462M     0  462M   0% /dev/shm
tmpfs           462M  6.3M  456M   2% /run
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
tmpfs           462M     0  462M   0% /sys/fs/cgroup
/dev/mmcblk0p1   42M   21M   21M  51% /boot

No sign of it. So lets find out where this bastard is. The 'lsblk' command will do this.

pi@raspberrypi:~ $ lsblk
NAME        MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda           8:0    0  3.7T  0 disk 
└─sda1        8:1    0  3.7T  0 part 
mmcblk0     179:0    0  7.5G  0 disk 
├─mmcblk0p2 179:2    0  7.4G  0 part /
└─mmcblk0p1 179:1    0 41.7M  0 part /boot

Why hello there, this system sees my drive on sda, and it being formatted and partitioned to 1 partition at sda1. Nice let's create that directory for it to mount to.

pi@raspberrypi:~ $ mkdir drive_caddy

Now here we have a folder called 'drive_caddy', we can only refer to it as that from the folder we are in. So it's absolute location is:

pi@raspberrypi:~ $ $ cd drive_caddy/
pi@raspberrypi:~/drive_caddy $ pwd
/home/pi/drive_caddy

So cd is 'change directory' and is how you move into the folders, next pwd is 'print working directory' which will show the absolute location from the directory you are in.

So when mounting sda1 to /home/pi/drive_caddy you can use the command as such:

pi@raspberrypi:~/drive_caddy $ sudo mount /dev/sda1 /home/pi/drive_caddy/
pi@raspberrypi:~/drive_caddy $ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/root       7.3G  910M  6.1G  13% /
devtmpfs        458M     0  458M   0% /dev
tmpfs           462M     0  462M   0% /dev/shm
tmpfs           462M  6.3M  456M   2% /run
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
tmpfs           462M     0  462M   0% /sys/fs/cgroup
/dev/mmcblk0p1   42M   21M   21M  51% /boot
/dev/sda1       3.6T  102M  3.4T   1% /home/pi/drive_caddy

And there she be, ready to go. But this is not the best way to do it for the following reasons:
  1. On reboot this mount will not happen on it's own
  2. We may not always get sda as the drive or sda1 as the partition
So we need to add it into a table of drives and partitions and directories that will mount automatically on boot. We also need to refer to the drive in a more consistent way and for this I am gonna use the drive UUID. This is a unique identifier for every drive in existence, and we find what this is for our drive like so:

pi@raspberrypi:~/drive_caddy $ sudo blkid /dev/sda1 -sUUID -ovalue
aa103a5c-ae3e-453b-81f8-5cbe5e777197

Sweet like chocolate, so now let enter that, and the other bollocks, into the table to mount at boot which is called fstab

pi@raspberrypi:~/drive_caddy $ sudo nano /etc/fstab

  GNU nano 2.2.6                            File: /etc/fstab                                                      Modified  

proc            /proc           proc    defaults          0       0
PARTUUID=eca6abc8-01 /boot           vfat    defaults          0       2
PARTUUID=eca6abc8-02  /               ext4    defaults,noatime  0       1
# a swapfile is not a swap partition, no line here
#   use  dphys-swapfile swap[on|off]  for that
UUID=aa103a5c-ae3e-453b-81f8-5cbe5e777197       /home/pi/drive_caddy/   ext4    defaults        0       1

Then 'ctrl-x' start the exit and 'y' to save changes. 


Ok let do some tests. First I am gonna unmount the drive with the 'umount' command on the drive, then mount everything from fstab with 'mount -a' and then check it's worked with 'df -h'. Then the ultimate by an actual reboot.

pi@raspberrypi:~/drive_caddy $ sudo umount /dev/sda1
pi@raspberrypi:~/drive_caddy $ sudo mount -a
pi@raspberrypi:~/drive_caddy $ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/root       7.3G  910M  6.1G  13% /
devtmpfs        458M     0  458M   0% /dev
tmpfs           462M     0  462M   0% /dev/shm
tmpfs           462M  6.3M  456M   2% /run
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
tmpfs           462M     0  462M   0% /sys/fs/cgroup
/dev/mmcblk0p1   42M   21M   21M  51% /boot
/dev/sda1       3.6T  102M  3.4T   1% /home/pi/drive_caddy
pi@raspberrypi:~/drive_caddy $ sudo reboot

Broadcast message from pi@raspberrypi on pts/0 (Fri 2017-07-28 18:55:49 UTC):

The system is going down for reboot NOW!

Back on and logged in and drive is there and mounted in the directory I created. Pukka.

Sharing is Caring

Now we need to access this directory from somewhere else, to view on a computer or external player like Kodi on that other Raspberry Pi you bought. So I am gonna have a simple SMB share over the network (at a later date will implement other sharing methods). Install SMB to the Pi to start with (press 'y' and 'enter' to confirm):

pi@raspberrypi:~ $ sudo apt-get install samba samba-common-bin

Now we to config the SMB sharing, on an app or program you would normally go into settings and fill out some boxes and check some other ones, but in this world we put all of those things in a text file which is at /etc/samba/smb.conf, we edit like before with nano. 

pi@raspberrypi:~ $ sudo nano /etc/samba/smb.conf 

First change this line to 'yes'

wins support = yes

Next we need to add a share, this is what directory is available to the network. I this case the drive that has been mounted. Stick this at the end of the file.

[pidow] 
   comment= Pidow the Pi Downloader 
   path=/home/pi/drive_caddy
   browseable=Yes
   writeable=Yes
   only guest=no
   create mask=0777
   directory mask=0777
   public=no 

Then ctrl-x and y to exit again. Now we are not going to be up and running straight away, we have to set up a password, enter the following command and then a password twice. Something not too obvious though.

pi@raspberrypi:~ $ sudo smbpasswd -a pi
New SMB password:
Retype new SMB password:
Added user pi.

To test I am gonna create a file and put it in the directory and see if I can view it on my computer over the network.

pi@raspberrypi:~ $ cd drive_caddy/
pi@raspberrypi:~/drive_caddy $ touch test.txt 



Files are sharing on the network, noice.


Big Downloady Bastard Time

So if you wanna leave it here and start downloading shit then there is a guide to wget command that will download from links. Here is an example of that process, I am gonna download big buck bunny from http://download.blender.org/peach/bigbuckbunny_movies/big_buck_bunny_480p_surround-fix.avi to the folder /home/pi/drive_caddy. Use -P after the link and before the directory.

pi@raspberrypi:~ $ wget http://download.blender.org/peach/bigbuckbunny_movies/big_buck_bunny_480p_surround-fix.avi -P /home/pi/drive_caddy/



Now gonna play this off my computer. 


Look at that glorious fucking bunny.

But we can be a bit more smarter about this, Aria2 is an application that will download link, and torrents, and a bunch of other stuff. So let's install this motherfucker.

pi@raspberrypi:~ $ sudo apt-get install aria2

Now time to edit config file, but this time we have to create our own. If nano can't see the file it will create it so we can use as before.

pi@raspberrypi:~ $ mkdir /home/pi/.aria2 pi@raspberrypi:~ $ nano .aria2/aria2.conf

I have added the following settings

continue
daemon=true
dir=/home/pi/drive_caddy
file-allocation=falloc
log-level=warn
max-connection-per-server=4
max-concurrent-downloads=3
max-overall-download-limit=200K
min-split-size=5M
enable-http-pipelining=true

enable-rpc=true
rpc-listen-all=true

rpc-secret=raspberry

If you are doing I would change the rpc-secret to something not so obvious. Apply the config with:

pi@raspberrypi:~ $ aria2c --conf-path=/home/pi/.aria2/aria2.conf

Much like the drive mounting we have to make sure it work on booting up. So create this file with nano

pi@raspberrypi:~ $ sudo nano /etc/init.d/aria2

We need to give the pi instructions on how to get everything up and running on boot, this is what the below code does.

#! /bin/sh
# /etc/init.d/aria2
### BEGIN INIT INFO
# Provides:          aria2
# Required-Start:    $local_fs
# Required-Stop:     $local_fs
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: aria2
### END INIT INFO

CONF_PATH=/home/pi/.aria2/aria2.conf

case "$1" in
    start)
        echo "Starting aria2 with custom configuration"
        aria2c --conf-path=$CONF_PATH
        ;;
    stop)
        echo "Stopping aria2"
        killall aria2c
        ;;
    *)
        echo "Usage: /etc/init.d/aria2 {start|stop}"
        exit 1
        ;;
esac

exit 0

Lastly we install the startup process by making the script runnable and then pushing the instruction over.

pi@raspberrypi:~ $ sudo chmod 0755 /etc/init.d/aria2

pi@raspberrypi:~ $ sudo update-rc.d aria2 defaults

I've gone and got a link and can now use this:

pi@raspberrypi:~/drive_caddy $ aria2c http://spumonte.com/files/PDFs/_Magazines_/2010%2007/Computer%20Music%20Magazine%20-%2007%202010.pdf



Sorted!

Where to go now?

So this is the first addition for now but have much more to come. Anything interesting to say then let me know @the1kingdom on Twitter.


Edits:

1. Totally fucked up location of the conf file
2. Also messed up with the init.d for startup

Comments

Popular posts from this blog

Raspberry Pi Download Machine Part 2

Well SSH is Boring I have got my RPi download machine up and running and having success, unfortunately though SSH is annoying and tedious, so I am gonna do something about it. But a disclaimer; this is my first time working with Aria2 in the most part. RPC  Remote Procedure Call is fancy way of saying 'make this thing make that thing do a thing'. So I ran the command to load the config file (and added to my init file) in the last post and ..... BOLLOCKS ALL. So ran the command again, and need to check the processes (like Task Manager). pi@raspberrypi : ~ $  ps aux | grep aria2c root       524  0.0  0.9  17028  9284 ?        Ss   21:21   0:00 aria2c --conf-path=/home/pi/.aria2/aria2.conf pi        1008  0.0  0.2   4272  1948 pts/0    S+   21:34   0:00 grep --color=auto aria2c There it is, that little scumbag. But still no dice. In the conf file I stated that the rpc-port to be 6800 so need to check that port.  A quick note on ports, every service

Adventures with Pillow Part 1

I am in the middle of building an app and need to put together several promo videos for Twitter and Instagram, and man it is boring. So I thought 'how can I automate this process?', so welcome to this post. What I am going for is a selection of images, greyscaled flipping through with our logo on the front. So my approach was to use python, pillow and ffmpeg to build something. This could be disastrous. Images To start a few pre-flight checks, a variable for the test image and to check for/create a processed folder for the final images to go. import os , errno from PIL import Image , ImageFont , ImageDraw testImage = 'test.png' cwd = os.getcwd() image_dir = os.path.join(cwd , 'images' ) default_dir = os.path.join(cwd , 'processed' ) try : os.makedirs(default_dir) except OSError as e: if e.errno != errno.EEXIST: raise I am going to build a function that will do all of the processing, but first going to run on one image b