Monday, January 30, 2006
FreeBSD Networking over FireWire
You might be familiar with Apple's implementation of IP over FireWire. This allows connecting two computers directly over FireWire ports. FreeBSD offers two drivers that provide networking over FireWire. fwe is a non-standard protocol, but it is implemented by default in the GENERIC kernel. fwip implements RFC 2734 (IPv4 over IEEE 1394) and RFC 3146 (Transmission of IPv6 Packets over IEEE 1394 Networks); it is available via kernel module.
I decided to have my laptop orr talk to my server janney using FireWire. To implement FireWire, orr uses an Adaptec DuoConnect PC Card Adapter and janney uses an Adaptec DuoConnect PCI Adapter. Both provide FireWire and USB 2.0.
Each system is running FreeBSD 6.0.
The laptop dmesg sees the following when the FireWire adapter is inserted.
The server dmesg sees the following at boot.
Here is the fwe interface that is created when the FireWire adapter is active on orr.
Here is the fwe interfaced that is created when the FireWire adapter is active on janney.
When I plug a FireWire cable into each host, dmesg on orr sees the following.
Server janney sees similar messages.
First I assign an IP to fwe0 on orr.
Next I assign an IP to fwe0 on janney.
Now the two systems can communicate over FireWire.
One can sniff the fwe0 interface with Tcpdump.
You can see that the fwe driver supports IPv6. The last four packets are IPv4.
Next I decided to try the fwip driver. I loaded the fwip kernel module on each system. First, orr.
This created the fwip0 interface, which I gave an IP address.
Notice dmesg output for orr.
Now, janney.
Notice janney's dmesg.
Now the two systems can communicate using the fwip driver.
You can also sniff fwip0. Take a look at the differences.
The first reports "Ethernet" because fwe is an Ethernet emulation layer. The second reports Apple IP-over-IEEE 1394 because that is an entirely new protocol. Check it out in Tethereal.
I noticed each host reported error messages like the following. I believe these were caused by the fwip driver.
I believe fwip has a future going forward, since it implements a standard.
For a rough idea of how fast these interfaces were, I transferred the same 100 MB file using the fwe, fwip, and fxp/xl wired Ethernet interfaces. Here are the results.
All the results are roughly the same, so the bottleneck is probably the hard drive of the laptop.
This opens some interesting possibilities for networking. Maybe I will buy a FireWire hub and connect multiple machines simultaneously.
I hope to try some of the debugging and memory access features of FireWire in the future.
Comments: You might be familiar with Apple's implementation of IP over FireWire. This allows connecting two computers directly over FireWire ports. FreeBSD offers two drivers that provide networking over FireWire. fwe is a non-standard protocol, but it is implemented by default in the GENERIC kernel. fwip implements RFC 2734 (IPv4 over IEEE 1394) and RFC 3146 (Transmission of IPv6 Packets over IEEE 1394 Networks); it is available via kernel module.I decided to have my laptop orr talk to my server janney using FireWire. To implement FireWire, orr uses an Adaptec DuoConnect PC Card Adapter and janney uses an Adaptec DuoConnect PCI Adapter. Both provide FireWire and USB 2.0.
Each system is running FreeBSD 6.0.
The laptop dmesg sees the following when the FireWire adapter is inserted.
cardbus0: CIS pointer is 0!
cardbus0: Resource not specified in CIS: id=10, size=800
cardbus0: Resource not specified in CIS: id=14, size=4000
fwohci0:mem 0x88004000-0x880047ff,0x88000000-0x88
003fff irq 11 at device 0.0 on cardbus0
fwohci0: OHCI version 1.10 (ROM=1)
fwohci0: No. of Isochronous channels is 4.
fwohci0: EUI64 08:00:28:56:02:00:49:8a
fwohci0: Phy 1394a available S400, 3 ports.
fwohci0: Link S400, max_rec 2048 bytes.
firewire0:on fwohci0
fwe0:on firewire0
if_fwe0: Fake Ethernet address: 0a:00:28:00:49:8a
fwe0: Ethernet address: 0a:00:28:00:49:8a
fwe0: if_start running deferred for Giant
sbp0:on firewire0
fwohci0: Initiate bus reset
fwohci0: node_id=0xc000ffc0, gen=1, CYCLEMASTER mode
firewire0: 1 nodes, maxhop <= 0, cable IRM = 0 (me)
firewire0: bus manager 0 (me)
cardbus0: CIS pointer is 0!
cardbus0: Resource not specified in CIS: id=10, size=1000
ohci0:mem 0x88000000-0x88000fff irq 11 at device
0.4 on cardbus0
ohci0: [GIANT-LOCKED]
usb1: OHCI version 0.0
usb1: unsupported OHCI revision
ohci0: USB init failed
The server dmesg sees the following at boot.
ohci0:mem 0xf7fff000-0xf7ffffff irq 22 at device
8.0 on pci4
ohci0: [GIANT-LOCKED]
usb1: OHCI version 1.0
usb1:on ohci0
usb1: USB revision 1.0
uhub1: NEC OHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub1: 3 ports with 3 removable, self powered
ohci1:mem 0xf7ffe000-0xf7ffefff irq 21 at device
8.1 on pci4
ohci1: [GIANT-LOCKED]
usb2: OHCI version 1.0
usb2:on ohci1
usb2: USB revision 1.0
uhub2: NEC OHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub2: 2 ports with 2 removable, self powered
ehci0:mem 0xf7ffdc00-0xf7ffdcff irq 20 at d
evice 8.2 on pci4
ehci0: [GIANT-LOCKED]
usb3: EHCI version 0.95
usb3: companion controllers, 3 ports each: usb1 usb2
usb3:on ehci0
usb3: USB revision 2.0
uhub3: NEC EHCI root hub, class 9/0, rev 2.00/1.00, addr 1
uhub3: 5 ports with 5 removable, self powered
fwohci0:mem 0xf7ffd000-0xf7ffd7ff,0xf7ff8000-0xf7
ffbfff irq 22 at device 12.0 on pci4
fwohci0: OHCI version 1.10 (ROM=1)
fwohci0: No. of Isochronous channels is 4.
fwohci0: EUI64 00:50:42:b5:c0:0d:0d:af
fwohci0: Phy 1394a available S400, 3 ports.
fwohci0: Link S400, max_rec 2048 bytes.
firewire0:on fwohci0
fwe0:on firewire0
if_fwe0: Fake Ethernet address: 02:50:42:0d:0d:af
fwe0: Ethernet address: 02:50:42:0d:0d:af
fwe0: if_start running deferred for Giant
sbp0:on firewire0
fwohci0: Initiate bus reset
fwohci0: node_id=0xc800ffc1, gen=1, CYCLEMASTER mode
firewire0: 2 nodes, maxhop <= 1, cable IRM = 1 (me)
firewire0: bus manager 1 (me)
Here is the fwe interface that is created when the FireWire adapter is active on orr.
fwe0: flags=108802mtu 1500
options=8
ether 0a:00:28:00:49:8a
ch 1 dma -1
Here is the fwe interfaced that is created when the FireWire adapter is active on janney.
fwe0: flags=108943mtu 1500
options=8
inet6 fe80::50:42ff:fe0d:daf%fwe0 prefixlen 64 scopeid 0x2
inet 10.1.1.2 netmask 0xffffff00 broadcast 10.1.1.255
ether 02:50:42:0d:0d:af
ch 1 dma 0
When I plug a FireWire cable into each host, dmesg on orr sees the following.
fwohci0: BUS reset
fwohci0: node_id=0xc800ffc1, gen=2, CYCLEMASTER mode
firewire0: 2 nodes, maxhop <= 1, cable IRM = 1 (me)
firewire0: bus manager 1 (me)
firewire0: New S400 device ID:005042b5c00d0daf
Server janney sees similar messages.
fwohci0: BUS reset
fwohci0: node_id=0x8800ffc0, gen=4, non CYCLEMASTER mode
firewire0: 2 nodes, maxhop <= 1, cable IRM = 1
firewire0: bus manager 1
firewire0: New S400 device ID:080028560200498a
First I assign an IP to fwe0 on orr.
orr:/home/richard$ sudo ifconfig fwe0 inet 10.1.1.1 netmask 255.255.255.0 up
orr:/home/richard$ ifconfig fwe0
fwe0: flags=108943mtu 1500
options=8
inet6 fe80::800:28ff:fe00:498a%fwe0 prefixlen 64 scopeid 0x4
inet 10.1.1.1 netmask 0xffffff00 broadcast 10.1.1.255
ether 0a:00:28:00:49:8a
ch 1 dma 0
Next I assign an IP to fwe0 on janney.
janney:/home/richard$ sudo ifconfig fwe0 inet 10.1.1.2 netmask 255.255.255.0 up
janney:/home/richard$ ifconfig fwe0
fwe0: flags=108943mtu 1500
options=8
inet6 fe80::50:42ff:fe0d:daf%fwe0 prefixlen 64 scopeid 0x2
inet 10.1.1.2 netmask 0xffffff00 broadcast 10.1.1.255
ether 02:50:42:0d:0d:af
ch 1 dma 0
Now the two systems can communicate over FireWire.
janney:/home/richard$ ping 10.1.1.1
PING 10.1.1.1 (10.1.1.1): 56 data bytes
64 bytes from 10.1.1.1: icmp_seq=0 ttl=64 time=0.694 ms
64 bytes from 10.1.1.1: icmp_seq=1 ttl=64 time=0.333 ms
64 bytes from 10.1.1.1: icmp_seq=2 ttl=64 time=0.312 ms
^C
--- 10.1.1.1 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max/stddev = 0.312/0.446/0.694/0.175 ms
One can sniff the fwe0 interface with Tcpdump.
07:36:42.479420 IP6 fe80::50:42ff:fe0d:daf > ff02::1:ff0d:daf: HBH ICMP6, multicast listener
reportmax resp delay: 0 addr: ff02::1:ff0d:daf, length 24
07:36:42.479667 IP6 fe80::50:42ff:fe0d:daf > ff02::2:f2f0:bec6: HBH ICMP6, multicast listener
reportmax resp delay: 0 addr: ff02::2:f2f0:bec6, length 24
07:36:42.479818 arp who-has 10.1.1.2 tell 10.1.1.2
07:36:42.498496 IP6 :: > ff02::1:ff0d:daf: ICMP6, neighbor solicitation, who has
fe80::50:42ff:fe0d:daf, length 24
07:36:43.756817 IP6 fe80::50:42ff:fe0d:daf > ff02::1:ff0d:daf: HBH ICMP6, multicast listener
reportmax resp delay: 0 addr: ff02::1:ff0d:daf, length 24
07:36:50.962703 IP6 fe80::50:42ff:fe0d:daf > ff02::2:f2f0:bec6: HBH ICMP6, multicast listener
reportmax resp delay: 0 addr: ff02::2:f2f0:bec6, length 24
07:36:54.422718 arp who-has 10.1.1.1 tell 10.1.1.2
07:36:54.422793 arp reply 10.1.1.1 is-at 0a:00:28:00:49:8a
07:36:54.423012 IP 10.1.1.2 > 10.1.1.1: ICMP echo request, id 34306, seq 0, length 64
07:36:54.423050 IP 10.1.1.1 > 10.1.1.2: ICMP echo reply, id 34306, seq 0, length 64
You can see that the fwe driver supports IPv6. The last four packets are IPv4.
Next I decided to try the fwip driver. I loaded the fwip kernel module on each system. First, orr.
orr:/home/richard$ sudo kldload fwip
orr:/home/richard$ kldstat
Id Refs Address Size Name
1 12 0xc0400000 63072c kernel
2 2 0xc0a31000 74b0 snd_csa.ko
3 3 0xc0a39000 1d408 sound.ko
4 1 0xc0a57000 c3a4 r128.ko
5 2 0xc0a64000 eeec drm.ko
6 16 0xc0a73000 568dc acpi.ko
7 1 0xc2236000 5000 if_fwip.ko
This created the fwip0 interface, which I gave an IP address.
orr:/home/richard$ ifconfig fwip0
fwip0: flags=108802mtu 1500
lladdr 8.0.28.56.2.0.49.8a.a.2.ff.fe.0.0.0.0
orr:/home/richard$ sudo ifconfig fwip0 inet 172.16.1.1 netmask 255.255.255.0 up
orr:/home/richard$ ifconfig fwip0
fwip0: flags=108843mtu 1500
inet6 fe80::a00:2856:200:498a%fwip0 prefixlen 64 scopeid 0x5
inet 172.16.1.1 netmask 0xffffff00 broadcast 172.16.1.255
lladdr 8.0.28.56.2.0.49.8a.a.2.ff.fe.0.0.0.0
Notice dmesg output for orr.
fwip0:on firewire0
fwip0: Firewire address: 08:00:28:56:02:00:49:8a @ 0xfffe00000000, S400, maxrec 2048
Now, janney.
janney:/home/richard$ sudo kldload fwip
janney:/home/richard$ kldstat
Id Refs Address Size Name
1 5 0xc0400000 641298 kernel
2 16 0xc0a42000 568dc acpi.ko
3 1 0xc214a000 5000 if_fwip.ko
janney:/home/richard$ ifconfig fwip0
fwip0: flags=108802mtu 1500
lladdr 0.50.42.b5.c0.d.d.af.a.2.ff.fe.0.0.0.0
janney:/home/richard$ sudo ifconfig fwip0 inet 172.16.1.2 netmask 255.255.255.0 up
janney:/home/richard$ ifconfig fwip0
fwip0: flags=108843mtu 1500
inet6 fe80::250:42b5:c00d:daf%fwip0 prefixlen 64 scopeid 0x6
inet 172.16.1.2 netmask 0xffffff00 broadcast 172.16.1.255
lladdr 0.50.42.b5.c0.d.d.af.a.2.ff.fe.0.0.0.0
Notice janney's dmesg.
fwip0:on firewire0
fwip0: Firewire address: 00:50:42:b5:c0:0d:0d:af @ 0xfffe00000000, S400, maxrec 2048
Now the two systems can communicate using the fwip driver.
janney:/home/richard$ ping -c 1 172.16.1.1
PING 172.16.1.1 (172.16.1.1): 56 data bytes
64 bytes from 172.16.1.1: icmp_seq=0 ttl=64 time=0.733 ms
--- 172.16.1.1 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max/stddev = 0.733/0.733/0.733/0.000 ms
You can also sniff fwip0. Take a look at the differences.
orr:/home/richard$ tcpdump -n -r fwe0.short.lpc -c 2 icmp
reading from file fwe0.short.lpc, link-type EN10MB (Ethernet)
07:36:54.423012 IP 10.1.1.2 > 10.1.1.1: ICMP echo request, id 34306, seq 0, length 64
07:36:54.423050 IP 10.1.1.1 > 10.1.1.2: ICMP echo reply, id 34306, seq 0, length 64
orr:/home/richard$ tcpdump -n -r fwip0.short.lpc -c 2 icmp
reading from file fwip0.short.lpc, link-type APPLE_IP_OVER_IEEE1394 (Apple IP-over-IEEE 1394)
07:41:52.593940 IP 172.16.1.2 > 172.16.1.1: ICMP echo request, id 38914, seq 0, length 64
07:41:52.593970 IP 172.16.1.1 > 172.16.1.2: ICMP echo reply, id 38914, seq 0, length 64
The first reports "Ethernet" because fwe is an Ethernet emulation layer. The second reports Apple IP-over-IEEE 1394 because that is an entirely new protocol. Check it out in Tethereal.
Frame 8 (102 bytes on wire, 102 bytes captured)
Arrival Time: Jan 30, 2006 07:41:52.593940000
Time delta from previous packet: 0.000231000 seconds
Time since reference or first frame: 13.327087000 seconds
Frame Number: 8
Packet Length: 102 bytes
Capture Length: 102 bytes
Protocols in frame: ap1394:ip:icmp:data
Apple IP-over-IEEE 1394, Src: 005042B5C00D0DAF, Dst: 80B2FFC100000000
Destination: 80B2FFC100000000
Source: 005042B5C00D0DAF
Type: IP (0x0800)
Internet Protocol, Src: 172.16.1.2 (172.16.1.2), Dst: 172.16.1.1 (172.16.1.1)
Version: 4
Header length: 20 bytes
Differentiated Services Field: 0x00 (DSCP 0x00: Default; ECN: 0x00)
0000 00.. = Differentiated Services Codepoint: Default (0x00)
.... ..0. = ECN-Capable Transport (ECT): 0
.... ...0 = ECN-CE: 0
Total Length: 84
Identification: 0x0696 (1686)
Flags: 0x00
0... = Reserved bit: Not set
.0.. = Don't fragment: Not set
..0. = More fragments: Not set
Fragment offset: 0
Time to live: 64
Protocol: ICMP (0x01)
Header checksum: 0x19f0 [correct]
Good: True
Bad : False
Source: 172.16.1.2 (172.16.1.2)
Destination: 172.16.1.1 (172.16.1.1)
Internet Control Message Protocol
Type: 8 (Echo (ping) request)
Code: 0
Checksum: 0xe88d [correct]
Identifier: 0x9802
Sequence number: 0x0000
Data (56 bytes)
0000 43 de 09 a3 00 09 3e e2 08 09 0a 0b 0c 0d 0e 0f C.....>.........
0010 10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e 1f ................
0020 20 21 22 23 24 25 26 27 28 29 2a 2b 2c 2d 2e 2f !"#$%&'()*+,-./
0030 30 31 32 33 34 35 36 37 01234567
I noticed each host reported error messages like the following. I believe these were caused by the fwip driver.
fwohci0: txd err= 3 miss Ack err
I believe fwip has a future going forward, since it implements a standard.
For a rough idea of how fast these interfaces were, I transferred the same 100 MB file using the fwe, fwip, and fxp/xl wired Ethernet interfaces. Here are the results.
- fwe: 104857600 bytes received in 00:09 (10.16 MB/s)
- fwip: 104857600 bytes received in 00:09 (11.01 MB/s)
- fxp/xl through a switch: 104857600 bytes received in 00:08 (11.21 MB/s)
- fxp/xl via crossover cable: 104857600 bytes received in 00:09 (10.75 MB/s)
All the results are roughly the same, so the bottleneck is probably the hard drive of the laptop.
This opens some interesting possibilities for networking. Maybe I will buy a FireWire hub and connect multiple machines simultaneously.I hope to try some of the debugging and memory access features of FireWire in the future.
To get some idea of peak TCP throughput, use iperf which is in ports.
ISTR tests on a mac using firewire which topped out at around 200 meg/sec.
Greg
ISTR tests on a mac using firewire which topped out at around 200 meg/sec.
Greg
Interesting, but I'm struggling with why you'd want to do this? Performance is similar to Ethernet, so why not just use the far more ubiquitous Ethernet?
I tried this technology because "it was there," like Everest. I'd like to know if anyone has any production uses?
I actually used it a few weeks ago, to install FreeBSD over the net on a machine where the onboard NIC wasn't supported by the installer, but the firewire port was. I set up bridging between default NIC and fwe0 on a helper machine, which allowed the new machine to install over the firewire port.
I have already used firewire as my 2nd NIC on my laptop when using FreeBSD which is always nice to have a choice.
Having in mind that firewire is still under GIANT it would be interesting to know the perfomance numbers you would get testing just the network component like Greg sugested.
Post a Comment
Having in mind that firewire is still under GIANT it would be interesting to know the perfomance numbers you would get testing just the network component like Greg sugested.
No comments:
Post a Comment
Note: Only a member of this blog may post a comment.