All content modules missing suddenly

Ina · April 6, 2019, 3:01pm

Hi there…

We brought another Rachel to Tanzania and it worked fine for a short while and then shortly after we left, the team reported that there were no modules listed and they went into he admin interface and nothing could be loaded.

I think the database is not loading based on the logs I can see… see below, any suggestions?

Sat Apr 6 15:50:12 2019 daemon.err /etc/init.d/mysql[3870]: WARNING: /.data/RACHEL/mysql not mounted… retrying in 3 secs.
Sat Apr 6 15:50:09 2019 daemon.err /etc/init.d/mysql[3859]: WARNING: /.data/RACHEL/mysql not mounted… retrying in 3 secs.
Sat Apr 6 15:50:06 2019 daemon.err /etc/init.d/mysql[3849]: WARNING: /.data/RACHEL/mysql not mounted… retrying in 3 secs.
Sat Apr 6 15:50:03 2019 daemon.err /etc/init.d/mysql[3839]: WARNING: /.data/RACHEL/mysql not mounted… retrying in 3 secs.
Sat Apr 6 15:50:01 2019 daemon.notice nodogsplash[3211]: Authenticating 192.168.88.130 98:9e:63:26:95:62
Sat Apr 6 15:50:01 2019 authpriv.info CRON[3815]: pam_unix(cron:session): session closed for user root
Sat Apr 6 15:50:01 2019 cron.info CRON[3815]: (CRON) info (No MTA installed, discarding output)
Sat Apr 6 15:50:01 2019 cron.info CRON[3816]: (root) CMD (/media/RACHEL/awstats/tools/awstats_updateall.pl now -awstatsprog=/media/RACHEL/awstats/wwwroot/cgi-bin/awstats.pl > /dev/null)
Sat Apr 6 15:50:01 2019 authpriv.info CRON[3815]: pam_unix(cron:session): session opened for user root by (uid=0)
Sat Apr 6 15:50:00 2019 daemon.err /etc/init.d/mysql[3811]: WARNING: /.data/RACHEL/mysql not mounted… retrying in 3 secs.
Sat Apr 6 15:49:58 2019 daemon.notice nodogsplash[3211]: Adding 192.168.88.130 98:9e:63:26:95:62 token 845a8d56 to client list
Sat Apr 6 15:49:57 2019 daemon.err /etc/init.d/mysql[3798]: WARNING: /.data/RACHEL/mysql not mounted… retrying in 3 secs.
Sat Apr 6 15:49:56 2019 daemon.info dnsmasq-dhcp[3555]: DHCPACK(br-lan) 192.168.88.130 98:9e:63:26:95:62 Allans-iMac
Sat Apr 6 15:49:56 2019 daemon.info dnsmasq-dhcp[3555]: DHCPREQUEST(br-lan) 192.168.88.130 98:9e:63:26:95:62
Sat Apr 6 15:49:55 2019 daemon.info dnsmasq-dhcp[3555]: DHCPOFFER(br-lan) 192.168.88.130 98:9e:63:26:95:62
Sat Apr 6 15:49:55 2019 daemon.info dnsmasq-dhcp[3555]: DHCPDISCOVER(br-lan) 98:9e:63:26:95:62
Sat Apr 6 15:49:55 2019 daemon.info dnsmasq-dhcp[3555]: DHCPOFFER(br-lan) 192.168.88.130 98:9e:63:26:95:62
Sat Apr 6 15:49:55 2019 daemon.info dnsmasq-dhcp[3555]: DHCPDISCOVER(br-lan) 98:9e:63:26:95:62
Sat Apr 6 15:49:54 2019 daemon.err /etc/init.d/mysql[3788]: WARNING: /.data/RACHEL/mysql not mounted… retrying in 3 secs.
Sat Apr 6 15:49:51 2019 daemon.err /etc/init.d/mysql[3776]: WARNING: /.data/RACHEL/mysql not mounted… retrying in 3 secs.
Sat Apr 6 15:49:48 2019 daemon.err /etc/init.d/mysql[3766]: WARNING: /.data/RACHEL/mysql not mounted… retrying in 3 secs.
Sat Apr 6 15:49:45 2019 daemon.err /etc/init.d/mysql[3756]: WARNING: /.data/RACHEL/mysql not mounted… retrying in 3 secs.
Sat Apr 6 15:49:42 2019 daemon.err /etc/init.d/mysql[3744]: WARNING: /.data/RACHEL/mysql not mounted… retrying in 3 secs.
Sat Apr 6 15:49:40 2019 daemon.warn dnsmasq[3555]: ignoring nameserver 127.0.0.1 - local interface
Sat Apr 6 15:49:40 2019 daemon.info dnsmasq[3555]: using local addresses only for domain lan
Sat Apr 6 15:49:40 2019 daemon.info dnsmasq[3555]: using nameserver 168.95.1.1#53
Sat Apr 6 15:49:40 2019 daemon.info dnsmasq[3555]: reading /tmp/resolv.conf
Sat Apr 6 15:49:39 2019 daemon.info dnsmasq-dhcp[3555]: read /etc/ethers - 0 addresses
Sat Apr 6 15:49:39 2019 daemon.info dnsmasq[3555]: read /tmp/hosts/dhcp - 1 addresses
Sat Apr 6 15:49:39 2019 daemon.err dnsmasq[3555]: failed to load names from /etc/hosts: Permission denied
Sat Apr 6 15:49:39 2019 daemon.warn dnsmasq[3555]: ignoring nameserver 127.0.0.1 - local interface
Sat Apr 6 15:49:39 2019 daemon.info dnsmasq[3555]: using local addresses only for domain lan
Sat Apr 6 15:49:39 2019 daemon.info dnsmasq[3555]: using nameserver 168.95.1.1#53
Sat Apr 6 15:49:39 2019 daemon.info dnsmasq[3555]: reading /tmp/resolv.conf
Sat Apr 6 15:49:39 2019 daemon.info dnsmasq[3555]: using local addresses only for domain lan
Sat Apr 6 15:49:39 2019 daemon.info dnsmasq[3555]: using nameserver 168.95.1.1#53
Sat Apr 6 15:49:39 2019 daemon.info dnsmasq-dhcp[3555]: DHCP, IP range 192.168.88.100 – 192.168.88.250, lease time 2h
Sat Apr 6 15:49:39 2019 daemon.info dnsmasq-dhcp[3555]: DHCP, IP range 203.30.1.100 – 203.30.1.109, lease time 2h
Sat Apr 6 15:49:39 2019 daemon.info dnsmasq[3555]: compile time options: IPv6 GNU-getopt DBus i18n IDN DHCP DHCPv6 no-Lua TFTP conntrack ipset auth DNSSEC loop-detect inotify
Sat Apr 6 15:49:39 2019 daemon.info dnsmasq[3555]: started, version 2.75 cachesize 150
Sat Apr 6 15:49:39 2019 daemon.info dnsmasq[2564]: exiting on receipt of SIGTERM
Sat Apr 6 15:49:39 2019 daemon.warn dnsmasq[2564]: ignoring nameserver 127.0.0.1 - local interface
Sat Apr 6 15:49:39 2019 daemon.info dnsmasq[2564]: using local addresses only for domain lan
Sat Apr 6 15:49:39 2019 daemon.info dnsmasq[2564]: using nameserver 168.95.1.1#53
Sat Apr 6 15:49:39 2019 daemon.info dnsmasq[2564]: reading /tmp/resolv.conf
Sat Apr 6 15:49:39 2019 daemon.notice nodogsplash[3211]: Waiting for connections
Sat Apr 6 15:49:39 2019 daemon.err /etc/init.d/mysql[3330]: WARNING: /.data/RACHEL/mysql not mounted… retrying in 3 secs.
Sat Apr 6 15:49:39 2019 daemon.notice nodogsplash[3211]: Initializing firewall rules
Sat Apr 6 15:49:38 2019 daemon.notice nodogsplash[3211]: Created web server on 192.168.88.1:2050
Sat Apr 6 15:49:38 2019 daemon.notice nodogsplash[3211]: Detected gateway br-lan at 192.168.88.1 (505BC20999B7)
Sat Apr 6 15:49:38 2019 daemon.notice nodogsplash[3211]: Initializing signal handlers
Sat Apr 6 15:49:38 2019 daemon.notice nodogsplash[3211]: Reading and validating configuration file /tmp/etc/nodogsplash_cfg0221da.conf
Sat Apr 6 15:49:38 2019 daemon.notice nodogsplash[2910]: Handler for termination caught signal 15
Sat Apr 6 15:49:36 2019 daemon.err /etc/init.d/mysql[3083]: WARNING: /.data/RACHEL/mysql not mounted… retrying in 3 secs.
Sat Apr 6 15:49:35 2019 daemon.notice nodogsplash[2910]: Waiting for connections
Sat Apr 6 15:49:34 2019 daemon.notice nodogsplash[2910]: Initializing firewall rules
Sat Apr 6 15:49:34 2019 daemon.notice nodogsplash[2910]: Created web server on 192.168.88.1:2050
Sat Apr 6 15:49:34 2019 daemon.notice nodogsplash[2910]: Detected gateway br-lan at 192.168.88.1 (505BC20999B7)
Sat Apr 6 15:49:34 2019 daemon.notice nodogsplash[2910]: Initializing signal handlers
Sat Apr 6 15:49:34 2019 daemon.notice nodogsplash[2910]: Reading and validating configuration file /tmp/etc/nodogsplash_cfg0221da.conf
Sat Apr 6 15:49:34 2019 daemon.notice nodogsplash[2198]: Handler for termination caught signal 15
Sat Apr 6 15:49:33 2019 user.notice firewall: Reloading firewall due to ifup of lan (br-lan)
Sat Apr 6 15:49:33 2019 daemon.err /etc/init.d/mysql[2751]: WARNING: /.data/RACHEL/mysql not mounted… retrying in 3 secs.
Sat Apr 6 15:49:32 2019 daemon.warn dnsmasq[2564]: ignoring nameserver 127.0.0.1 - local interface
Sat Apr 6 15:49:32 2019 daemon.info dnsmasq[2564]: using local addresses only for domain lan
Sat Apr 6 15:49:32 2019 daemon.info dnsmasq[2564]: using nameserver 168.95.1.1#53
Sat Apr 6 15:49:32 2019 daemon.info dnsmasq[2564]: reading /tmp/resolv.conf
Sat Apr 6 15:49:32 2019 daemon.info dnsmasq-dhcp[2564]: read /etc/ethers - 0 addresses
Sat Apr 6 15:49:32 2019 daemon.info dnsmasq[2564]: read /tmp/hosts/dhcp - 1 addresses
Sat Apr 6 15:49:32 2019 daemon.err dnsmasq[2564]: failed to load names from /etc/hosts: Permission denied
Sat Apr 6 15:49:32 2019 daemon.warn dnsmasq[2564]: ignoring nameserver 127.0.0.1 - local interface
Sat Apr 6 15:49:32 2019 daemon.info dnsmasq[2564]: using local addresses only for domain lan

jeremy · April 6, 2019, 9:02pm

Hi @Ina — this isn’t totally abnormal behaviour, usually a result of holding the power button down to shut off the device. The device needs to be reset by tapping three times or using the software shutdown button.

It should be easy to solve if you just let the device sit on for like 15 minutes after rebooting, the modules should appear if you refresh the page.

Ina · April 7, 2019, 2:17pm

Thanks Jeremy,

I tried the 3 taps and sw shutdown, left the device off. restarted and and let it run for 15-20 minutes and still no modules.

I didn’t actually know that the power button shouldn’t be used to turn off the device by holding it, is that the same for all Rachels?

Any suggestions?

Ina/Allan

jeremy · April 10, 2019, 1:25am

@steve and I are looking into this.

Steve · April 10, 2019, 5:10pm

Hi Ina,

Very sorry to hear of the difficulty you are having with the unit. I will look into this issue and expect to provide an answer/solution tomorrow.

With regard to power off. Holding down the power button till power off is called an ungraceful shutdown. This method should not be used on any Rachel Plus device. The multi-quick-tap method (or using the power off button in admin->hardware tab).

(Note: Realizing potential effects of an ungraceful shutdown may be more prevalent on hard drives manufactured over the past couple years as a result of background tasks and aerial density increases.)

Ina · April 18, 2019, 7:20pm

Thanks Steve, any update?

Steve · April 24, 2019, 6:50pm

Hi Ina,

I believe I recall you knowing how to ssh into Rachel. If so:

In a terminal on a separate device (assuming you are connected via Wi-Fi to the Rachel Plus device), type: ssh root@192.168.88.1
Enter pswd, type: (contact us)
Once on the Rachel command line, type: fsck -y /dev/sda1
Reboot the device. Wait 5 minutes and load the Rachel home page.

I will provide a link to a procedure shortly, in the event that you (or others on this form) are unfamiliar with using ssh to log on to the Rachel Plus command line.

andal03 · April 27, 2019, 8:08pm

I must be doing something wrong… Here is the readut:
root@192.168.88.1’s password:
Welcome to CMAL100

140 packages can be updated.
96 updates are security updates.

*** /dev/mmcblk0p2 will be checked for errors at next reboot ***

Last login: Sat Apr 27 20:51:05 2019 from 192.168.88.130
root@CMAL-a788:~# fsck -y /dev/sda1
fsck from util-linux 2.27.1
e2fsck 1.42.13 (17-May-2015)
fsck.ext2: No such file or directory while trying to open /dev/sda1
Possibly non-existent device?
root@CMAL-a788:~#

Steve · April 29, 2019, 6:29pm

Hi Ina,

It looks like you did it right. Unfortunately, this may not be the simple fix we were looking to leverage. The first thing to address is the message: *** /dev/mmcblk0p2 will be checked for errors at next reboot ***

I would recommend rebooting with graceful shutdown, ssh in again and try the command again. If you get the same result (1) please send the output again if any different at all (2) perform the following:

This command will show us if the hard drive is alive and available. Type: blkid | grep sda
(do not add the “1” at the end of sda this time). If the device is available, you should see: /dev/sda1: UUID=“some number representing the device uuid…” If there is no output, it means the hard drive was not detected. If there is an output, please send the output.

Sincerely,

Steve

airsay · May 1, 2019, 5:12pm

Waoh! Password in the clear!!!

moderator edit; thanks for catching

andal03 · May 11, 2019, 12:23pm

Hi Steve,

Sorry for taking so long…,

It looks like hard drive is dead… Here are the output:

Last login: Sat Apr 27 21:01:29 2019 from 192.168.88.130
root@CMAL-a788:~# fsck -y /dev/sda1
fsck from util-linux 2.27.1
e2fsck 1.42.13 (17-May-2015)
fsck.ext2: No such file or directory while trying to open /dev/sda1
Possibly non-existent device?
root@CMAL-a788:~# blkid | grep sda
root@CMAL-a788:~#

What is the next step?

Allan

Steve · May 11, 2019, 5:36pm

Hi Allan,

Please provide your order number and last 4-digits of the MAC Address printed on the back label. I will look into the order and see what can be done to help resolve… There is an associated modules list as well that I can provide at a minimum.

At a high level, how to restore the unit to working level:

Swap the hard drive (I think there is a video or instructions on our form. I’ll check. But removing 7 screws on the back and flat-head screwdriver to remove the back plate. From there it is a straight shot to removing the HDD (7 screws). Be sure to reuse all of the assembly, including the rubber grommets. Caution, it is easy to drop a screw into the unit. It can be hard getting it back out. Proceed with cation.
On the command line type: /sbin/datadiskchk.sh /dev/sda (this formats and partitions the drive as sda1, required for content)
Perform a graceful reboot (tap power button 3x)
Load modules:

Utility to load modules from a USB device:
ftp://ftp.worldpossible.org/other/MultiModLoad.zip
Procedure to load modules from a USB device:
ftp://ftp.worldpossible.org/other/Rachel_Plus_Multi_Mod_Load_Procedure.pdf

Steve

andal03 · May 12, 2019, 4:40pm

Hi Steve,

The order # was 1814 and the last of the MAC 0d166. It is an 1TB unit.

If you remember, the original purchase was when there was the issue with Kolibri and that the units wouldnt turn WIFI back on. So, this was eventually sent back somewhere around October 11th 2018. And while it worked here in the US it stopped working after a few days in Tanzania.

I am assuming that I can load the modules over the internet as well, correct?

Let us know what can be done…

Allan

jeremy · May 15, 2019, 4:59pm

2 posts were split to a new topic: Cannot Connect to RACHEL 3.0 Server

Steve · May 17, 2019, 4:01pm

Hi Allan,

Yes. Modules can be downloaded via internet from http://oer2go.org/
Use the ‘Install’ tab in the admin area. I will follow up via email with the modules list.

Steve