Baryon

From Yocto Project
Jump to navigationJump to search

Hardware: Black Sand

  • N450 1.6G Hz CPU
  • 1GB Memory
  • 3 160GB SATA hard disks
  • 2 NICs: Intel 82567v-3 and 82574L Gigabit Ethernet

Time Frame and Schedule

Oct. 12 - Oct. 28 

 Oct. 12 - Oct. 14: Proposal ready and agreement
 Oct. 15 - Oct. 25: Key services ready
 Oct. 25 - Oct. 28: Profiling and tracing

Purpose

  • Create a NAS layer and build NAS solution with basic features on Intel hardware with Yocto shortly;
  • Compare with commercial NAS products for covering those key services which commercial products have in the exercise;
  • Performance study on the key services and debugging with ADT Eclipse plugins;
  • Verify Yocto usability, as well as ADT tracing tools, and file bugs if we notice any difficulties to use Yocto;
  • Improve its performance built with Yocto is “nice to have”, not P1 priority.

Features and Assignments

  • Samba Server (Edwin)
  • Web Interface (Paul)
  • NFS and AFS (Ke)
  • Media Tomb (Dongxiao)
  • HTTP and FTP (Dexuan)
  • SSH and RSYNC (Dexuan)
  • RAID (Dexuan)
  • Profiling and Tracing (Dongxiao, Lianhao and Jiajun)

Nice to Have

  • AFS
  • HTTP and FTP
  • Hardware RAID

Some of Commercial Products in China Market

  • QNAP TS459-Pro+ (~$1300)
  • QNAP TS409-Pro (~$500)
  • TS419P (~$700)
  • Synology DS411+ (~$800)

Web Interface Candidates

webmin; ajenti; freenas; openfiler.

Feature Details

Discovery and Detection

For the first connection of a NAS device, there must be an application provided by the vendor on CD to detect the IP address of the NAS device to configurate, and later on, users can change the settings on the Web administrative interface. But we don't have this kind of tool in Yocto so we have to fix the IP address of Black Sand unless a display connected to the NAS device is allowed. The walk-around Paul suggested is to use avahi and set a reasonably unique hostname. and ensure avahi is running on startup then it should be visible as <hostname>.local for those machines that support bonjour/mdns/etc. lookups.

Here is the investigation result on some of the commercial products done by Shane:

There must be a client software in CD and that is used to find NASes which are not configured in the network at the first time. Later on, the web interface takes effect.

Software Installation After installing the NAS hardware, proceed to the software installation. The following demonstration is based on Windows OS.

1. Install the QNAP Finder from the product CD-ROM.

2. Run the Finder. If the Finder is blocked by your firewall, unblock the utility.

3. The Finder detects the NAS which has not been configured. Click “Yes” to perform quick setup of the NAS.

When first powered on, during the initial boot sequence, the DNS-323 will wait to be assigned an IP address via DHCP. If it does not receive a DHCP assigned IP address, by default it will have a self-assigned IP address of 192.168.0.32. It is recommended to use the included D-Link Easy Search Utility when accessing and configuring the DNS-323 for the first time. If you want to change the IP address before logging in or are unable to connect to the DNS-323 IP address, you can use the Easy Search Utility provided on the product CD to locate the device on your network and make any needed changes.

Samba

  • Ported samba recipes from OE (Edwin)
  • Setup the local webmin server at very early stage (Jiajun)
  • Test how to integrate samba service with webmin in the Yocto image compared with the local webmin server at early stage. (Jiajun and Edwin)

Webmin

  • Ported webmin (Paul)
  • Packaged webmin-module-* packages (Paul)
    • webmin-module-proc
    • webmin-module-mount
    • webmin-module-fdisk
    • webmin-module-raid
    • webmin-module-samba
    • webmin-module-proftpd
    • webmin-module-exports
    • webmin-module-init
    • webmin-module-useradmin
    • webmin-module-time
    • webmin-module-servers
    • webmin-module-net (Paul and Dexuan)
    • webmin-module-lvm (Dexuan, needed by RAID)
  • Enabled dropbear SSH server (Paul)
  • Edwin to cover Paul on webmin during ELC-E

NFS

  • AFS is dropped.
  • Enabled NFS options in kernel config (Ke)
  • Added NFS admin utility (Ke)
  • Enabled/Verified webmin panel to configurate NFS (Ke)
  • Enabled NFS service "Restart" after settings are changed (Ke)

Media Tomb

  • Added media tomb (Dongxiao)
  • Added external link into webmin since media tomb has its own web interface (Jiajun)

RAID

  • Hard RAID is dropped.
  • Added Soft RAID and enabled in webmin panel. (Dexuan)
  • Tested RAID performance(Dexuan)

SSH and RSYNC

  • Already there, verified (Dexuan)

FTP

  • http is dropped.
  • Added proftpd and enabled in webmin panel (Dexuan)

Profiling and Tracing with ADT

  • Netperf Testing (Dongxiao)
  • File Transfer Testing (Dongxiao)
  • IOSTAT Testing (Dongxiao)
  • blktrace Profiling (Dongxiao)
  • Perf Testing (Dongxiao)
  • Oprofile and Iozone (Lianhao)
Netperf Testing
 Environment
 NAS Server: Blacksand.
 Client: Yocto-build1. CPU: Intel(R) Core(TM) i7 CPU 860 @ 2.80GHz (4 Core, 8 thread). Memory: 4GB. Harddisk: 1T.
 
 Netperf Testing
 Description:
 - Netperf: NAS Server
 - Netserver: Yocto-build1
 - Command: netperf –H 192.168.79.73 –c –C
 - Send CPU is the CPU utilization when sending packets, ditto for recv CPU.
 Results summary:
 - The throughput is near the network bandwidth.
 Netperf.png
File Transfer Testing
 Description:
 - Test file download (from NAS server to client) speed on different protocols.
 Results summary:
 - For FTP and NFS, the speed is also limited by the hard disk speed on the receiver side, therefore 57M and 63M do make sense.
 - For SAMBA service, Dongxiao tried with other two samba servers (without NAS solution) and their bandwidths are also around 20M.
 Filetransfer.png
IOSTAT Testing
 Description:
 - Issue FTP download, SAMBA copy, and NFS copy simultaneously, and run iostat in NAS server.
 - All these services are run on a soft raid /dev/md0, which contains /dev/sdb and /dev/sdc
 Results summary: 
 - This is a profiling result, showing the TPS (Transactions Per Second), read/write speed, and some system information.
 Iostat.png
blktrace Profiling
 Description:
 - Issue FTP download, SAMBA copy, and NFS copy simultaneously, and run blktrace in NAS server.
 - Command: blktrace -d /dev/md0 -o - | blkparse -i –
 Results summary:
 - Summary Result is showed as follows.
 - Detailed profiling result contains specific operations performed upon each I/O submitted into the block I/O layer.
 
 CPU0 (9,0):
 Reads Queued:       33459,     6259MiB  Writes Queued:           0,        0KiB
 Read Dispatches:        0,        0KiB  Write Dispatches:        0,        0KiB
 Reads Requeued:         0               Writes Requeued:         0
 Reads Completed:        0,        0KiB  Writes Completed:        0,        0KiB
 Read Merges:            0,        0KiB  Write Merges:            0,        0KiB
 Read depth:             0               Write depth:             0
 IO unplugs:             0               Timer unplugs:           0
 CPU1 (9,0):
 Reads Queued:       33523,     6236MiB  Writes Queued:           0,        0KiB
 Read Dispatches:        0,        0KiB  Write Dispatches:        0,        0KiB
 Reads Requeued:         0               Writes Requeued:         0
 Reads Completed:        0,        0KiB  Writes Completed:        0,        0KiB
 Read Merges:            0,        0KiB  Write Merges:            0,        0KiB
 Read depth:             0               Write depth:             0
 IO unplugs:             0               Timer unplugs:           0
 
 Total (9,0):
 Reads Queued:       66982,    12495MiB  Writes Queued:           0,        0KiB
 Read Dispatches:        0,        0KiB  Write Dispatches:        0,        0KiB
 Reads Requeued:         0               Writes Requeued:         0
 Reads Completed:        0,        0KiB  Writes Completed:        0,        0KiB
 Read Merges:            0,        0KiB  Write Merges:            0,        0KiB
 IO unplugs:             0               Timer unplugs:           0
 
 Throughput (R/W): 0KiB/s / 0KiB/s
 Events (9,0): 66982 entries
 Skips: 0 forward (0 -   0.0%)
Perf Testing

1. Bench testing. 

 Description:
 - Perf is a performance analysis tool based on CPU/PMU/software counter/tracepoints.
 - Command: perf bench all
 Result Summary:
 - Test subsystem like scheduler, IPC mechanism, and memory access performance.
 
 # Running sched/messaging benchmark...
 # 20 sender and receiver processes per group
 # 10 groups == 400 processes run
 
    Total time: 3.569 [sec]
 
 # Running sched/pipe benchmark...
 # Executed 1000000 pipe operations between two tasks
 
    Total time: 58.444 [sec]
 
     58.444584 usecs/op
         17110 ops/sec
 
 # Running mem/memcpy benchmark...
 # Copying 1MB Bytes ...
 
    287.769784 MB/Sec
      1.119911 GB/Sec (with prefault)

2. perf top 

 Description: 
 - Issue FTP download, SAMBA copy, and NFS copy simultaneously, and run “perf top” in NAS server.
 - The perftop is something like oprofile, which can track the system hotspot for developers to narrow down performance issues.
 
    PerfTop:    2022 irqs/sec  kernel:97.5%  exact:  0.0% [1000Hz cycles],  (all, 2 CPUs)
 ------------------------------------------------------------------------------------------  
            samples  pcnt function                      DSO
            _______ _____ _____________________________ ___________________________________
 
             823.00 10.0% acpi_pm_read                  /boot/vmlinux-3.0.4-yocto-standard+
             279.00  3.4% sub_preempt_count             /boot/vmlinux-3.0.4-yocto-standard+
             244.00  3.0% debug_smp_processor_id        /boot/vmlinux-3.0.4-yocto-standard+
             215.00  2.6% add_preempt_count             /boot/vmlinux-3.0.4-yocto-standard+
             210.00  2.6% _raw_spin_lock_irqsave        /boot/vmlinux-3.0.4-yocto-standard+
             178.00  2.2% schedule                      /boot/vmlinux-3.0.4-yocto-standard+
             163.00  2.0% acpi_os_read_port             /boot/vmlinux-3.0.4-yocto-standard+
             161.00  2.0% svc_recv                      /boot/vmlinux-3.0.4-yocto-standard+
             140.00  1.7% get_parent_ip                 /boot/vmlinux-3.0.4-yocto-standard+
             112.00  1.4% sched_clock_local             /boot/vmlinux-3.0.4-yocto-standard+
             100.00  1.2% select_task_rq_fair           /boot/vmlinux-3.0.4-yocto-standard+
              96.00  1.2% in_lock_functions             /boot/vmlinux-3.0.4-yocto-standard+
              95.00  1.2% __copy_from_user_ll_nocache   /boot/vmlinux-3.0.4-yocto-standard+
              94.00  1.1% _raw_spin_unlock_irqrestore   /boot/vmlinux-3.0.4-yocto-standard+
              90.00  1.1% e1000_xmit_frame              /boot/vmlinux-3.0.4-yocto-standard+
              86.00  1.0% __rcu_read_unlock             /boot/vmlinux-3.0.4-yocto-standard+
              85.00  1.0% acpi_idle_enter_bm            /boot/vmlinux-3.0.4-yocto-standard+
              78.00  0.9% leave_mm                      /boot/vmlinux-3.0.4-yocto-standard+
              73.00  0.9% __copy_user_intel             /boot/vmlinux-3.0.4-yocto-standard+
              71.00  0.9% reschedule_interrupt          /boot/vmlinux-3.0.4-yocto-standard+
              68.00  0.8% sched_clock                   /boot/vmlinux-3.0.4-yocto-standard+
              66.00  0.8% memcpy                        /boot/vmlinux-3.0.4-yocto-standard+
              65.00  0.8% update_curr                   /boot/vmlinux-3.0.4-yocto-standard+
              64.00  0.8% dequeue_task_fair             /boot/vmlinux-3.0.4-yocto-standard+
              63.00  0.8% __switch_to                   /boot/vmlinux-3.0.4-yocto-standard+
              63.00  0.8% page_address                  /boot/vmlinux-3.0.4-yocto-standard+
              62.00  0.8% svc_tcp_has_wspace            /boot/vmlinux-3.0.4-yocto-standard+
              62.00  0.8% __rcu_read_lock               /boot/vmlinux-3.0.4-yocto-standard+
              61.00  0.7% sched_clock_cpu               /boot/vmlinux-3.0.4-yocto-standard+
              60.00  0.7% mutex_spin_on_owner           /boot/vmlinux-3.0.4-yocto-standard+
              58.00  0.7% enqueue_task_fair             /boot/vmlinux-3.0.4-yocto-standard+
              58.00  0.7% local_bh_enable_ip            /boot/vmlinux-3.0.4-yocto-standard+
              56.00  0.7% _raw_spin_lock_bh             /boot/vmlinux-3.0.4-yocto-standard+
              55.00  0.7% tcp_write_xmit                /boot/vmlinux-3.0.4-yocto-standard+
              49.00  0.6% try_to_wake_up                /boot/vmlinux-3.0.4-yocto-standard+
              49.00  0.6% tcp_transmit_skb              /boot/vmlinux-3.0.4-yocto-standard+
              48.00  0.6% svc_xprt_enqueue              /boot/vmlinux-3.0.4-yocto-standard+
              48.00  0.6% get_page_from_freelist        /boot/vmlinux-3.0.4-yocto-standard+
              47.00  0.6% kfree                         /boot/vmlinux-3.0.4-yocto-standard+
              46.00  0.6% rcu_note_context_switch       /boot/vmlinux-3.0.4-yocto-standard+
              42.00  0.5% cpuidle_idle_call             /boot/vmlinux-3.0.4-yocto-standard+
              41.00  0.5% tcp_ack                       /boot/vmlinux-3.0.4-yocto-standard+
              40.00  0.5% default_send_IPI_mask_logical /boot/vmlinux-3.0.4-yocto-standard+
              38.00  0.5% ttwu_stat                     /boot/vmlinux-3.0.4-yocto-standard+
              38.00  0.5% getnstimeofday                /boot/vmlinux-3.0.4-yocto-standard+
              34.00  0.4% local_bh_disable              /boot/vmlinux-3.0.4-yocto-standard+
              33.00  0.4% acpi_hw_validate_io_request   /boot/vmlinux-3.0.4-yocto-standard+
              32.00  0.4% pick_next_task_fair           /boot/vmlinux-3.0.4-yocto-standard+
Oprofile and Iozone
 Desciption:
 Sheet nfs-iozone: iozone data for NFS service
 Sheet samba-iozone: iozone data for Samba service
 Sheet samba-oprofile: oprofile output when a client is coping a 4GB files through Samba service
 
 Write:          This test measures the performance of writing a new file.
 Re-write:       This test measures the performance of writing a file that already exists.
 Read:           This test measures the performance of reading an existing file.  
 Re-Read:        This test measures the performance of reading a file that was recently read.
 Random Read:    This test measures the performance of reading a file with accesses being made to random locations within the file.
 Random Write:   This test measures the performance of writing a file with accesses being made to random locations within the file.
 Backwards Read: This test measures the performance of reading a file backwards.
 Strided Read:   This test measures the performance of reading a file with a strided access behavior.
 
 The unit of the sizes is "bytes".
 Due to the record (block, cache or buffer),
 when the file is not large enough, the speed reflects the benifit of the records,
 but when the file size exceeds the record size, the speed reflects the real and is more stable for each case.
 
 Again, for Oprofile sighting, Oprofile exhauses CPU a bit, i.e., 13% out of all CPU unhalted resource.
 
 <Please open the attached results with Excel> File:Iozone.xls

Full Testing

  • Tested NAS features (Jiajun)

Feature Comparisons

We compared the features of our Baryon device with a high-end NAS device QNAP TS-459 Pro+ Turbo NAS and another D-Link DNS-323. The comparsion result is: Feature compare.png

Our Baryon device basically covered most of key services in NAS devices.

Known Issues

1. Oprofile causes kernel panic with heavy workload of files transferred via ftp or samba When profiling and tracing with Oprofile, copying a big file (say a 4GB file) causes kernel hangs. The procedures to reproduce are: 

$opcontrol --start-daemon --separate=lib -c 6
$opcontrol --start

Then on a client, copying a 4GB file from NAS, NAS system will hang. Without Oprofile, the error has never happened.

2. For NFS, in Webmin configuration pages, exporting NFS directories with NFS 4.0 protocol doesn't work, i.e., clients could not mount those directories. However, with NFS 3.0 protocol, it does work.

Difficulties

From Lianhao:

  • Oprofile will cause kernel hang when profiling.
  • There is no kernel remote debug tool in ADT. All debug tools are for application debugging.
  • Webmin is not easy to operate, the configuration pages are too complex. For example, NFS, users (even ourselves) should have knowledges of NFS before configurating NFS.
  • Services should start up in a specific order. E.g., RAID must be started first before NFS or Samba, if the directory is on RAID. Users probably don't know that.

From Edwin:

  • The poky master is not stable, say, before porting samba from oe, setting up the environment costs much time.

From Dongxiao:

  • The error message is not helpful. E.g., "no provider" stands for "commercial license" issue when building media tomb.
  • No maintainer for meta-demo, after gcc is upgraded, we should fix some bugs in meta-demo to make sure it can be compiled.

From Dexuan:

  • The terminal shell doesn't support Ctrl+Z, Ctrl+Z will cause hang, which is bad.

More detail information is: in sato's terminal, vi doesn't work well when Ctrl+Z is pressed: e.g., let me first try "vi /etc/hosts", then I want to put vi into background so I press Ctrl+Z and next I find I can't exit from vi -- this may be expected because IIRC by default sato busybox's shell doesn't support putting tasks background (why? This is a really useful feature), but what's worse is: I can't enter vi's command mode at all, e.g., vi doesn't execute commands like ":wq" and actually vi doesn't respond to the Escape key properly; what's the worst is: I can't quit the vi editor from the same terminal! As a final result, the terminal becomes actually unusable any longer and I have to open a new terminal and find the pid of vi and use 'kill' to exit the vi process... And actually this can be reproduced in qemu(e.g., qemux86 core-image-sato), but what surprised me is: qemux86 core-image-minimal doesn't have such an issue.

From Ke:

  • The kernel config manual is not straight forward. In order to find how to config the kernel, Ke spent much time digging out where and how in the documents.

From Jiajun:

  • Webmin package building is so slow. So is rebuilding. A question, can the task to find dependency be multi-threading?
  • For Baryon image, we include some tools for profiling etc. Can we have two images (one for debug and another smaller one for release), i.e., debugging tools can be removed for the release images.

From Paul:

  • Building packages such as Webmin that contain a lot of files but no binaries is unnecessarily slow. Filed bug #1677, have an initial improvement but work on package.bbclass is still needed.
  • Webmin's source contains files with names that include the * character, and under certain circumstances this interacts poorly with our packaging code. Filed bug #1676, worked around in the webmin recipe for the moment by renaming the errant files.
  • do_package_split() is very useful for packaging modules or plugins separately, however it does not seem to be documented and given that it not straightforward to use, it ought to be.
  • Building for qemu* targets always forces a build of X11 (bug #1690). I have a fix for the oprofileui-server half of this already.
  • We don't have sound enabled for qemux86, this would not be difficult to fix and would be useful for testing media players (bug #1683)
  • Packaging of mkfs within util-linux was not ideal for Webmin. I have now split this out to a separate package and the fix is in master.
  • We don't seem to have ext4 support enabled. Probably just a matter of enabling the right kernel module, but it seems we ought to enable it or document how to do so.
  • We may want to look into some mechanism of setting the web interface and/or root passwords for a real device, or at least think of how to advise people to do it in the documentation. Think about a device vendor who wants to have a sticky label on the case with the default password (which is randomly generated for each device). This might be something that gets done at the time the image is written to the device rather than when the image is built.

Code Repo

The code check-in is WIP. After that, the repo can be downloaded from http://git.yoctoproject.org/cgit/cgit.cgi/meta-baryon. But now, the code is in the contrib tree at http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=dcui/meta-baryon-v2.

Retrieved from "https://wiki.yoctoproject.org/wiki/index.php?title=Baryon&oldid=3958"