Interesting and understood - maybe I just need to document that step as I can see people moving content around on USB’s.
Interesting that you say /media/usb0, /media/usb1 , I’ve seen it go to /media/pi/xxx and read somewhere that this was the default, but maybe only default on the non-default version.
Are you saying that I could just do apt-get usbmount if reqd , or does it require configuration? I can’t try it now as my RPi is packed for the flight tomorrow.
The /media/pi path could be something done on the desktop versions of Raspbian. I’m not sure what they use for USB auto-mounting. /media/usb0, /media/usb1, etc, is specific to the usbmount package and you can install that with a sudo apt-get install usbmount. By default it supports the filesystems listed in /etc/usbmount/usbmount.conf and if you want to use other filesystems like exfat it requires installing extra packages and changing settings in that file. That’s where it gets messy with USB booting.
When I get my Raspberry Pi 4 I’ll probably release an experimental image with a lot of changes and redesigns I’ve been working on, including USB module loading and I’ll probably build a modified usbmount to fix these OS drive remounting issues.
I have the internet archive server running and after modifying the internetarchive.service file’s WorkingDirectory path and installing the service it runs at boot. I didn’t have to write anything else. I have a basic RACHEL module set up and everything is working so I’ll upload it tomorrow after I double check it’s all working automatically as part of the install script. For now I think people will have to install this themselves and if the install script is put up on github it should only be one command.
James
Thanks - I’ll test it when I get to SF (I arrive Tues night); I think installing from a script will work fine for now, if people start using it then its worth considering for part of the main installation - for now anyway I’m doing fast enough revisions that I want people to install from npm rather than an image so that they are running the current version. I’ll include the script on the dweb-mirror github so the instructions become really simple to fetch and run it.
Sounds good. This installer with files should work. It runs all of the commands, copies and adds the internetarchive.service to run at system boot, copies the module to the modules directory and sets ownership/permissions. I hope that work on your end.
I have some weird behaviour on the server’s end where it sends me to arc/archive.org/details/local which is a blank page the first time. After accessing archive.org/details/home that page loads fine with the UI and everything.
The module I made is very basic. It doesn’t show the status of the service and for the first 30 seconds or so after boot it’s not available. Is there a json link or something that I can add a check for to see if it’s up? The module is at /var/www/modules/en-internet_archive and the rachel-index.php can be edited to change the description etc.
James
Hi James, I’m arrived and getting into this …
I’ve got the downloader from your site to my Mac, but I’m unclear if this needs to be put somewhere specific before running it.
I’m just figuring out install instructions - which I’ll write up but I think they are in essence.
Download image from http://rachelfriends.org/downloads/public_ftp/rachelpi_64EN/rachelpi_2019/rachel-pi_kolibi_buster_unofficial.7z
Flash to MicroSD using Ester,
Install SD in Raspberry
curl https://drive.google.com/open?id=1wz8Z7Y_xLilTdgK47flXcWg4oEAKCkeV
unzip file downladed
cd into zip
./install.sh
is that your understanding
By the way … is there a reason you use the .7z format, that Etcher (balenaEtcher on macs) doesn’t recognize ? It recognizes zip files. Obviously this is solvable, but the instructions for .7z have to be a step more complex involving Terminal action … (Unzip it, then etch it)
Hi @mitra. Those instructions should work for you. You’ll want to put the unzipped folder into the /var/tmp folder and run the script with sudo permissions as the script itself isn’t written with sudo commands in it. “sudo sh install.sh” is what I do. I don’t intend on hosting the internetarchive.zip on my google drive. I was just putting it there for you to download and do what you want with. I think it would be better if you put those files on your github and used a git clone for it as part of the official instructions for other people as I will probably remove that from my google drive once you have it. You can then make changes and improve it from there.
I use 7zip because on ultra settings it compresses the images significantly more than zip which makes the downloads faster for slower connections and the uploads possible for me. I’ve never used it but https://www.keka.io/en/ apparently handles 7z on mac OSX. Etcher is the program to flash it and Balena acquired it recently so that’s why there is confusion around Etcher/BalenaEtcher. There is more info about installing and using RACHEL-Pi images in this post.
James
OK - will continue working my way through, I’ll move it to one of the Archive.org sites when its done, rather than git clone (which would make our dweb-mirror distro unneccessarily big, or require a separate distro)
Oops - I misread your message … -
yes, I can have the internetarchive install files on our distro,
For the main image, is that location stable-ish ?
That Raspbian Buster image is temporary for people with the pi4 to test and I figured it would be best to make this installer for it as moving forward everyone with new installations will be using Buster. I will be uploading new Buster based images once I get a pi4 and can test with it. It is on the World Possible FTP so it will be stable until I make new images some time this month ( hopefully ). The link address will change but there shouldn’t be any other changes required in terms of the script. I’ll test it when I make those though and let you know about any new links.
James
I think I’ve got it working one of my RPI’s though I need to test more in the morning, and find a place for the install script (I don’t want people to have to do a git clone).
I know I have some first-time-it-runs issues that I’ve flagged doing some test installs, I’ll work through them, it works but doesn’t create the best experience yet.
There is a json status curl http://localhost:4244/info gets some info about the connection and is what I use in the browser to check the local server is up.The “transportstatuses.http.status” in that JSON should be 0 when the local server can see the archive main server.
The files are up in https://github.com/internetarchive/dweb-mirror in the rachel subdirectory.
Great. I hope everything running as expected. I’ll look at adding the JSON check to the module.
James
Seems to run as expected, its not responding sometimes, and trying to figure out why, but that could be a fault in how I’m testing it.
I failed on getting it to read USB hard drives, I wanted to demo at DwebCamp with a 4TB drive full of content, but cant get it to work even after install usbmount One challenge is figuring out what format the drives are - I see logs about the failure, but nothing telling me which format to add to usbmount.conf. I’m pretty sure one of the drives I’m testing with will be the default mac FS format, and the other is default Windows (brand new Seagate drive).
I think the relevant part of the log is
Jul 14 13:12:46 rachel kernel: [ 3.103975] usb 1-1.2: New USB device found, idVendor=1058, idProduct=25e2, bcdDevice=40.05
Jul 14 13:12:46 rachel kernel: [ 3.103997] usb 1-1.2: New USB device strings: Mfr=2, Product=3, SerialNumber=1
Jul 14 13:12:46 rachel kernel: [ 3.104007] usb 1-1.2: Product: My Passport 25E2
Jul 14 13:12:46 rachel kernel: [ 3.104016] usb 1-1.2: Manufacturer: Western Digital
Jul 14 13:12:46 rachel kernel: [ 3.104025] usb 1-1.2: SerialNumber: 5758333144363843334B3932
Jul 14 13:12:46 rachel kernel: [ 3.105122] usb-storage 1-1.2:1.0: USB Mass Storage device detected
Jul 14 13:12:46 rachel kernel: [ 3.139611] scsi host0: usb-storage 1-1.2:1.0
Jul 14 13:12:46 rachel kernel: [ 3.521823] usb 1-1.1.2: new high-speed USB device number 6 using dwc_otg
Jul 14 13:12:46 rachel rsyslogd: imuxsock: Acquired UNIX socket '/run/systemd/journal/syslog' (fd 3) from systemd. [v8.1901.0]
Jul 14 13:12:46 rachel kernel: [ 3.645288] EXT4-fs (mmcblk0p2): re-mounted. Opts: (null)
Jul 14 13:12:46 rachel kernel: [ 3.732848] usb 1-1.1.2: New USB device found, idVendor=0bc2, idProduct=231a, bcdDevice= 7.10
Jul 14 13:12:46 rachel kernel: [ 3.732865] usb 1-1.1.2: New USB device strings: Mfr=1, Product=2, SerialNumber=3
Jul 14 13:12:46 rachel kernel: [ 3.732875] usb 1-1.1.2: Product: Expansion
Jul 14 13:12:46 rachel rsyslogd: [origin software="rsyslogd" swVersion="8.1901.0" x-pid="336" x-info="https://www.rsyslog.com"] start
Jul 14 13:12:46 rachel kernel: [ 3.732884] usb 1-1.1.2: Manufacturer: Seagate
Jul 14 13:12:46 rachel kernel: [ 3.732893] usb 1-1.1.2: SerialNumber: NAADTPQD
Jul 14 13:12:46 rachel kernel: [ 3.733677] usb 1-1.1.2: The driver for the USB controller dwc_otg_hcd does not support scatter-gather which is
Jul 14 13:12:46 rachel kernel: [ 3.733689] usb 1-1.1.2: required by the UAS driver. Please try an other USB controller if you wish to use UAS.
Jul 14 13:12:46 rachel kernel: [ 3.733702] usb-storage 1-1.1.2:1.0: USB Mass Storage device detected
Jul 14 13:12:46 rachel kernel: [ 3.734091] scsi host1: usb-storage 1-1.1.2:1.0
Jul 14 13:12:46 rachel kernel: [ 4.273746] scsi 0:0:0:0: Direct-Access WD My Passport 25E2 4005 PQ: 0 ANSI: 6
Jul 14 13:12:46 rachel kernel: [ 4.278704] sd 0:0:0:0: [sda] Spinning up disk...
Jul 14 13:12:46 rachel kernel: [ 4.281397] scsi 0:0:0:1: Enclosure WD SES Device 4005 PQ: 0 ANSI: 6
Jul 14 13:12:46 rachel kernel: [ 4.410937] usbcore: registered new interface driver uas
Jul 14 13:12:46 rachel kernel: [ 4.587486] media: Linux media interface: v0.10
If you have any insights.
I’m guessing the Seagate drive is the HFS+ one? The issue with the Seagate drive and Raspberry Pi is probably fixed by the steps in this post. HFS+ is supported by default by usbmount and you can see it in the /etc/usbmount/usbmount.conf as “hfsplus”.
The windows filesystem is probably exFat but may be NTFS. I would plug it into another device to check the file system. NTFS is supported but requires extra steps that are discussed in this post. exFat requires installing the exfat-fuse package with sudo apt-get install exfat-fuse and then changing the FILESYSTEMS in /etc/usbmount/usbmount.conf to add exfat and fuseblk. FILESYSTEMS=“vfat ext2 ext3 ext4 ntfs-3g ntfs exfat hfsplus fuseblk” for example.
James
I’m still drawing a blank - the USB stick doesn’t mount, though it mounts fine on IIAB which has essentially the same usbmount.conf,
The Seagate / NTFS drive wont mount on either IIAB or Rachel, according to /var/log/messages its seeing it on sda1 but tels me it “does not contain a filesystem or disklabel”. I reformatted it as exfat and there was no change.
Nor will the WD / hfsplus drive (which says it requires a force option as it doesnt support writing journaled drives - that’s fair enough, I don’t need to mount this drive, and was just using it for getting a different test point).
Note I’ve set max_usb_current=1 in /boot/config.txt as suggested in the article you linked to, and done apt-get install exfat-fuse
At this point I think I’m going to have to leave out this functionality until after the camp, unless someone has succeeded in getting either USB sticks or hard disks to mount on
I’ll try it today and see what happens. Maybe something has changed with Buster. I have had usbmount and exfat drives working on Raspbian Stretch Lite using the same process. Keep in mind these issues aren’t with RACHEL as it’s just a web server and doesn’t support USB content yet. This is to do with Linux and Raspbian. This post from the IIAB github mentions the same issue and links the same solution post.
Understood - its more of a challenge of whether we need to package away the geekiness to make things usable by mere earthlings, and maybe ours is the first server to have enough content to be worth bringing it in on an external drive ?
Note that IIAB has fixed the USB stick problem, since that post, but doesnt appear to have fixed issues with external hard drives (or at least, I’m hitting issues with two different drives).
Can you try changing the “MountFlags=slave” line in /lib/systemd/system/systemd-udevd.service to “MountFlages=shared” and see if that helps?
There is no "MountFlags=slave” lines, or anything similar.
Which section should MountFlags=shared be in [Unit] or [Service]
Looks like that has changed from previous Raspbians. Try changing “PrivateMounts=yes” from that file under service to “PrivateMounts=no”