MIPS/ERLite-3

From Gentoo Wiki
< MIPS
Jump to:navigation Jump to:search
Resources

The EdgeRouter Lite (ERLite-3) is an MIPS64 router (MIPS64r2, Cavium Octeon) with 512 MB of RAM, which uses a (removable) USB pendrive for storage. This document describes how to run Gentoo on this device.

EdgeRouter Lite original packaging
EdgeRouter Lite (inside)

What's is provided

GPL Sources

Ubiquiti provides a GPL archive containing the sources of all open-source software that runs on this router. The GPL archive can be found here [1]

The Processor

system type             : UBNT_E100 (CN5020p1.1-500-SCP)
processor               : 0
cpu model               : Cavium Octeon+ V0.1
BogoMIPS                : 1000.00
wait instruction        : yes
microsecond timers      : yes
tlb_entries             : 64
extra interrupt vector  : yes
hardware watchpoint     : yes, count: 2, address/irw mask: [0x0ffc, 0x0ffb]
isa                     : mips1 mips2 mips3 mips4 mips5 mips64r2
ASEs implemented        :
shadow register sets    : 1
kscratch registers      : 0
core                    : 0
VCED exceptions         : not available
VCEI exceptions         : not available

processor               : 1
cpu model               : Cavium Octeon+ V0.1
BogoMIPS                : 1000.00
wait instruction        : yes
microsecond timers      : yes
tlb_entries             : 64
extra interrupt vector  : yes
hardware watchpoint     : yes, count: 2, address/irw mask: [0x0ffc, 0x0ffb]
isa                     : mips1 mips2 mips3 mips4 mips5 mips64r2
ASEs implemented        :
shadow register sets    : 1
kscratch registers      : 0
core                    : 1
VCED exceptions         : not available
VCEI exceptions         : not available

The USB Flash Drive

The details of the USB flash drive that comes with this board are the following:

CODE USB Details
Bus 002 Device 012: ID 13fe:3e00 Kingston Technology Company Inc. Flash Drive

Partitioning

The USB flash drive on the board has two partitions. A small FAT32 one for the Linux Kernel image and a big ext3 one containing the root filesystem in a squashfs file. The squashfs file is in the GPL archive as well.

Note
Although it is possible to replace the supplied USB memory stick with another one, be careful as not all of them work. If you managed to find one that works please edit this page to list the 'lsusb -v' output for that USB flash drive. This is the reason why this guide only focuses on the nfsroot configuration. 
 Original USB stick
 ID 13fe:4100 Kingston Technology Company Inc
 
 Working replacements:


 Samsung 64GB USB 3.0 Flash Drive Fit (MUF-64BB/AM)
 https://www.amazon.com/gp/product/B013CCTNKU  
 ID 090c:1000 Silicon Motion, Inc. - Taiwan (formerly Feiya Technology Corp.)

 SanDisk Cruzer Fit 32GB USB 2.0 Low-Profile Flash Drive- SDCZ33-032G-B35
 http://www.amazon.com/gp/product/B00812F7O8
 ID 0781:5571 SanDisk Corp. Cruzer Fit
 
 Kingston Digital USB 3.0 DataTraveler Mini (Cyan) (DTM30/32GB)
 http://www.amazon.com/dp/B00GGD10ZE
 ID 0951:1666 Kingston Technology

 SanDisk Ultra Fit 128GB USB 3.0 Flash Drive (SDCZ43-128G-G46) 
 https://www.amazon.com/gp/product/B00YFI1EBC
 ** Note: you may need to prepend 'usb reset;sleep 3;' to the bootcmd to make this boot reliably !!
 
 Kingston Digital DataTraveler SE9 8GB USB 2.0 (DTSE9H/8GBZET)
 http://www.amazon.com/dp/B00DYQYJ3Q
 http://www.amazon.cn/dp/B006W8U2MU (Amazon.CN link for China users)
 scsi 0:0:0:0: Direct-Access     Kingston DataTraveler 2.0 1.00 PQ: 0 ANSI: 4
 sd 0:0:0:0: [sda] 15148608 512-byte logical blocks: (7.75 GB/7.22 GiB)
 sd 0:0:0:0: [sda] Write Protect is off
 sd 0:0:0:0: [sda] Mode Sense: 45 00 00 00
 sd 0:0:0:0: [sda] Write cache: disabled, read cache: enabled, doesn't support DP  O or FUA
  sda: sda1 sda2 sda3
 sd 0:0:0:0: [sda] Attached SCSI removable disk

The memory before block 2048 is nonempty. Use dd to copy the first 2048 blocks.

A more comprehensive and detailed guide for swapping the original USB flash drive could be found at this thread on Ubiquiti forum.
CODE Default USB partitioning
Disk /dev/sde: 3880 MB, 3880452096 bytes, 7579008 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x65690700

   Device Boot      Start         End      Blocks   Id  System
/dev/sde1            2048      292863      145408    c  W95 FAT32 (LBA)
/dev/sde2          292864     3710975     1709056   83  Linux

The Linux Kernel

The Linux kernel that comes with the board is a modified 2.6.32.13 one. If you ever want to rebuild it yourself from the GPL archive, you need a mips64-octeon-linux-gnu toolchain that comes with the Cavium SDK. Typical distro toolchains (mips64-unknonwn-*) will fail to compile these kernel sources with errors like these:

root #make ARCH=mips CROSS_COMPILE=mips64-unknown-linux-gnu-
Error: Opcode not supported on this processor: octeon (mips64r2) `saa $6,($7)
Error: Opcode not supported on this processor: octeon (mips64r2) `saa $9,($7)
Error: Opcode not supported on this processor: octeon (mips64r2) `saa $3,($7)

The RootFS

This router comes with Debian 6.0.6

FILE /etc/debian_version
6.0.6

Preparing the Gentoo MIPS64 RootFS

Getting the MIPS64 stage3 tarball

The ERLite-3 uses a 64-bit MIPS64r2 Big-Endian Cavium Octeon processor. So the stage3 we want for this board is a mip64-* one. For this guide, we will pick a glibc based one.

root #mkdir /mnt/erlite-3/
root #tar xvjf stage3-mips64_multilib-20130715.tar.bz2 -C /mnt/erlite-3
root #rm stage3-mips64_multilib-20130715.tar.bz2

Getting the latest portage snapshot

root #tar xjvpf portage-latest.tar.bz2 -C /mnt/erlite-3/usr
root #rm portage-latest.tar.bz2

Configure /etc/fstab

If you are going to use NFSroot to boot your ERLite-3 board, you need to edit the /dev/ROOT entry in your fstab like this

FILE /mnt/erlite-3/etc/fstab
<ip_of_nfs_server>:/mnt/erlite-3 / nfs rw,auto,async,tcp,noatime,vers=3  0 1

Reset root password

In order to be able to login after the first boot, you need to reset the root password. For this edit the /mnt/erlite-3/etc/shadow file and remove the '*' from the second column.

Important
Remember to set your root password once you have successfully boot your board for the first time.

Building the toolchain

The are many different ways to build a mips64 toolchain. We will use the Gentoo sys-devel/crossdev script to build one.

root #emerge --ask crossdev

Create a cross toolchain for MIPS64 (big-endian system):

root #crossdev -t mips64-unknown-linux-gnu

Make sure you read this crossdev info and this Embedded handbook to understand how to use crossdev in your day-to-day development.

The MIPS64 Linux Kernel

Getting the Kernel

We will use the Linux Kernel from the Linus' git repo for now, but there might be other repos more suitable for this board out there. If you find one, please edit this guide as appropriate.

root #mkdir /mnt/erlite-3-kernel ; cd /mnt/erlite-3-kernel
root #git clone --depth 1 git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
root #cd linux

Building the Kernel

root #ARCH=mips CROSS_COMPILE=mips64-unknown-linux-gnu- make cavium_octeon_defconfig
root #ARCH=mips CROSS_COMPILE=mips64-unknown-linux-gnu- make
root #ARCH=mips CROSS_COMPILE=mips64-unknown-linux-gnu- make modules_install INSTALL_MOD_PATH=/mnt/erlite-3/

Booting Up

NFS Boot

Install and Configure the TFTP Server

The TFTP server will be used to load the new kernel image on the board. For this, you need the net-ftp/tftp-hpa package.

root #emerge --ask tftp-hpa

Edit the /etc/conf.d/in.tftpd file as appropriate:

FILE /etc/conf.d/in.tftpd
# /etc/init.d/in.tftpd

# Path to server files from
# Depending on your application you may have to change this.
# This is commented out to force you to look at the file!
#INTFTPD_PATH="/var/tftp/"
#INTFTPD_PATH="/tftpboot/"
INTFTPD_PATH="/mnt/erlite-3/kernel/"

# For more options, see in.tftpd(8)
# -R 4096:32767 solves problems with ARC firmware, and obsoletes
# the /proc/sys/net/ipv4/ip_local_port_range hack.
# -s causes $INTFTPD_PATH to be the root of the TFTP tree.
# -l is passed by the init script in addition to these options.
INTFTPD_OPTS="-R 4096:32767 -s ${INTFTPD_PATH}"

And start the service (add it to the default runlevel if you wish)

root #rc-update add in.tftpd default (optional)
root #/etc/init.d/in.tftpd start

Install and Configure the NFS Server

The NFS Server will export the (previously extracted and prepared) stage3 (/mnt/erlite-3) to the ERLite-3 board.

In order to use your PC as an NFS server, you need the following kernel options to be enabled:

KERNEL 
File Systems --->
    Network File Systems --->
        <*> NFS server support
        -*-   NFS server support for NFS version 3

You also need to build and configure the net-fs/nfs-utils as follows:

root #emerge --ask nfs-utils

Then edit the /etc/exportfs as follows:

FILE /etc/exports
# /etc/exports: NFS file systems being exported.  See exports(5).
/mnt/erlite3/ 192.168.1.2(async,rw,no_root_squash,no_subtree_check)
root #rc-update add nfs default (optional)
root #/etc/init.d/nfs start

Prepare U-Boot for NFS boot

U-boot is pre-configured as follows:

CODE U-Boot configuration
bootdelay=0
baudrate=115200
download_baudrate=115200
nuke_env=protect off $(env_addr) +$(env_size);erase $(env_addr) +$(env_size)
autoload=n
ethact=octeth0
bootcmd=fatload usb 0 $loadaddr vmlinux.64;bootoctlinux $loadaddr coremask=0x3 root=/dev/sda2 rootdelay=15 rw rootsqimg=squashfs.img rootsqwdir=w mtdparts=phys_mapped_flash:512k(boot0),512k(boot1),64k@3072k(eeprom)
loadaddr=0x9f00000
numcores=2
stdin=serial
stdout=serial
stderr=serial
env_addr=0x1fbfe000
env_size=0x2000
flash_base_addr=0x1f800000
flash_size=0x400000
uboot_flash_addr=0x1f880000
uboot_flash_size=0x70000
flash_unused_addr=0x1f8f0000
flash_unused_size=0x310000
bootloader_flash_update=bootloaderupdate

For NFS boot, we need to set the following variables: 

ipaddr : IP for the ERLite-3 board (e.g. 192.168.1.2)
serverip : IP for the PC acting as tftp server (e.g. 192.168.1.3)
bootcmd : We need to override the existing 'bootcmd' command with one suitable for tftp boot

For this, use the following commands in the u-boot command line (Press Ctrl-C to interrupt the boot process) 

setenv ipaddr 192.168.1.2
setenv serverip 192.168.1.3
setenv bootcmd 'tftpboot $loadaddr vmlinux; bootoctlinux $loadaddr coremask=0x3 ip=192.168.1.2::192.168.1.254:255.255.255.0:edge:eth0:off root=/dev/nfs nfsroot=192.168.1.3:/mnt/erlite-3/,tcp,vers=3

And now save the new configuration 

saveenv

Now, it is time to reset the router and boot into your shiny new Gentoo MIPS64 rootfs.

CODE Serial output
This is localhost.unknown_domain (Linux mips64 3.9.0-rc7-00173-g830ac85-dirty) 00:00:21

 localhost login: root
 localhost ~ #

USB Boot

Successful boot from USB drive is confirmed on vanilla kernel 3.11_rc4, same is true fo Kernel 3.17. A kernel configuration file is available at https://github.com/xypron/kernel-edgerouter.

Important
There is still a high chance of kernel panic during SOFT reboots of device, if it is booted from USB, at least for kernel 3.11.*.
Important
Kernel 3.13 has some nasty bug in Octeon USB driver, details [here]

Console

CODE 
pinkbyte@octeon ~ $ uname -a

Related options in kernel config

user $pinkbyte@octeon ~ $ zcat /proc/config.gz | grep OCTEON
CONFIG_CAVIUM_OCTEON_SOC=y
# CONFIG_CAVIUM_OCTEON_2ND_KERNEL is not set
CONFIG_CAVIUM_OCTEON_CVMSEG_SIZE=2
CONFIG_CAVIUM_OCTEON_LOCK_L2=y
CONFIG_CAVIUM_OCTEON_LOCK_L2_TLB=y
CONFIG_CAVIUM_OCTEON_LOCK_L2_EXCEPTION=y
CONFIG_CAVIUM_OCTEON_LOCK_L2_LOW_LEVEL_INTERRUPT=y
CONFIG_CAVIUM_OCTEON_LOCK_L2_INTERRUPT=y
CONFIG_CAVIUM_OCTEON_LOCK_L2_MEMCPY=y
# CONFIG_OCTEON_ILM is not set
CONFIG_CPU_CAVIUM_OCTEON=y
CONFIG_SYS_HAS_CPU_CAVIUM_OCTEON=y
CONFIG_PATA_OCTEON_CF=y
CONFIG_OCTEON_MGMT_ETHERNET=y
CONFIG_MDIO_OCTEON=y
CONFIG_HW_RANDOM_OCTEON=y
CONFIG_I2C_OCTEON=y
CONFIG_SPI_OCTEON=y
CONFIG_OCTEON_WDT=y
CONFIG_USB_OCTEON_EHCI=y
CONFIG_USB_OCTEON_OHCI=y
CONFIG_USB_OCTEON2_COMMON=y
CONFIG_EDAC_OCTEON_PC=y
CONFIG_EDAC_OCTEON_L2C=y
CONFIG_EDAC_OCTEON_LMC=y
CONFIG_EDAC_OCTEON_PCI=y
CONFIG_OCTEON_ETHERNET=y
CONFIG_OCTEON_USB=y

Kernel command line parameters can be adjusted through U-boot.

CODE 
pinkbyte@octeon ~ $ cat /proc/cmdline
Retrieved from "https://wiki.gentoo.org/index.php?title=MIPS/ERLite-3&oldid=1291819"