Understanding USRP2 flow control

GNU Radio
1

Dear All!

I am starting to learn my new USRP2 device.
I bought a reasonable fast quad core PC and
a recommended ethernet board.

I wanted to find out if I can get the interpolation 4
working in my environment. I just connect a sine
oszillator to the usrp2 sink.

I am observing the signal on a connected scope.

Everything works fine for a while, but then suddenly
the output becomes heavily distorted.

I can see from the resource watch of the system
monitor that my PC putting out data at a sustained
rate of more than 800 Mbits/sec.

I started to try to understand the code in
gnuradio/usrp2/host/lib and expected some flow control
when the usrp2 input buffers are nearly full .
Unfortunately I was not able to understand how this
is implemented.

Is there any flow control at all?
If not, what is supposed to happen, if the PC is
sending (slightly) too fast?

Appreciating any information on this subject,
Roland

–

_ _ | Roland Schwarz OE1RSA
|)( | sip:[email protected]
| __) | mailto:[email protected]
________| http://www.blackspace.at

2

On Sun, Jul 18, 2010 at 06:50:47PM +0200, OE1RSA wrote:

I am observing the signal on a connected scope.
when the usrp2 input buffers are nearly full .
Unfortunately I was not able to understand how this
is implemented.

Is there any flow control at all?
If not, what is supposed to happen, if the PC is
sending (slightly) too fast?

Appreciating any information on this subject,
Roland

Flow control is handled at the link level using ethernet PAUSE frames.

Be sure to enable real-time scheduling by calling

if gr.enable_realtime_scheduling() != gr.RT_OK:
    print "Note: failed to enable realtime scheduling, continuing"

before starting or running the flow graph.

If you haven’t already, ensure that /etc/security/limits.conf contains
this line:

@usrp - rtprio 50

and that you are a member of group “usrp”.

[eb@octo ~]$ groups
eb wheel dialout usrp

Eric

3

Thank you for your answer.

Am 2010-07-19 22:06, schrieb Eric B.:

Flow control is handled at the link level using ethernet PAUSE frames.

I was not aware of this ethernet “feature”. Having learned
something new.

@usrp - rtprio 50

and that you are a member of group “usrp”.

Made everyting as told, but still no success.

Another observation I have made:

I am using two kernels from the ubuntu 10.04 LTS distro:

2.6.32 generic and
2.6.31 rt

It is interesting that the generic kernel does better than
the rt kernel. I.e. the rt is practically unusable at
interpolation 4. The generic does quite well for a while,
then suddenly drops off.

Btw.: I am using the e1000e driver for intel cards.

Possibly it may be an issue with the driver?

Do you have an idea how I could find out whether my
problems are because of buffer overrun or underrun?

Regards Roland

–

_ _ | Roland Schwarz OE1RSA
|)( | sip:[email protected]
| __) | mailto:[email protected]
________| http://www.blackspace.at

4

On Tue, Jul 20, 2010 at 10:24:51AM +0200, OE1RSA wrote:

Be sure to enable real-time scheduling by calling

and that you are a member of group “usrp”.

Made everyting as told, but still no success.

Just to be clear, do you get the

“Note: failed to enable realtime scheduling”

or not. I’m assuming that you’re not seeing this.

then suddenly drops off.

Btw.: I am using the e1000e driver for intel cards.

Possibly it may be an issue with the driver?

That driver should be fine.

If your machine has any kind of power control and/or frequency
scaling, be sure that it’s configured in its “Performance mode”. That
is, run at full speed all the time, do not try to save energy,
throttle the CPU, etc.

Be sure that flow control is properly configured.
It should look like this with a USRP2 plugged in and powered up:

$ ethtool -a eth1
Pause parameters for eth1:
Autonegotiate:  on
RX:             on
TX:             off

Do you have an idea how I could find out whether my
problems are because of buffer overrun or underrun?

On transmit, it’ll be a buffer underrun. The host isn’t keeping up.

You could hook up a USB CMOS/TTL to serial dongle and watch the
diagnostic
output from the USRP2’s serial port.

Eric

5

Hi Eric,

it took me some time to assemble the RS 232 level shifter.

First to explicitly answer your question about realtime
scheduling. Yes I did put in the explicit enabling you
have told me, and yes, I get no error, i.e. realtime scheduling
should be set succesfully.

2010-07-20 20:18, Blossom:

If your machine has any kind of power control and/or frequency
scaling, be sure that it’s configured in its “Performance mode”. That
is, run at full speed all the time, do not try to save energy,
throttle the CPU, etc.

I tried to tune my BIOS settings, but this did not change the
behaviour substantly.

Be sure that flow control is properly configured.
It should look like this with a USRP2 plugged in and powered up:

$ ethtool -a eth1
Pause parameters for eth1:
Autonegotiate:  on
RX:             on
TX:             off

Yes, this is how it looks on my machine.

I also experimentally set it to
Autonegotiate: off
RX: off
TX: off

to see, if it would change something, but as you guessed,
it did not improve the situation.

On transmit, it’ll be a buffer underrun. The host isn’t keeping up.

I can now confirm that I am seeing bursts of ‘UUU…’ on the debug
port of the USRP2.

Some more observations:

  1. The 6.31 rt kernel from the ubuntu 10.04 LTS is worse than
    the default stock 6.32 kernel.

  2. Altough the USRP2 is signaling underuns ( I guess this is what U
    stands for) at the same time I can see in wireshark that I am
    receiving PAUSE frames. This looks like there might be some buffer
    size problems. How large are the transmit buffers of the USRP2?

  3. I wrote a small test app that blasts out UDP frames on the same
    interface as fast as it can, and I can see a sustained rate of
    approx 900Mbits/ses. So I guess it is not a limitation of my
    hardware.

Do you have any other ideas what else I could try?

Turning to receive side: I invoked usrp2_fft.py -e eth2 -d 4
and I can see not see ‘S’ on the terminal. Am I correct in the
assumption this means I have no missed packets on receive?

Thank you for your help!
Roland

–

_ _ | Roland Schwarz OE1RSA
|)( | sip:[email protected]
| __) | mailto:[email protected]
________| http://www.blackspace.at

6

On Fri, Jul 23, 2010 at 10:35:12AM +0200, OE1RSA wrote:

Hi Eric,

it took me some time to assemble the RS 232 level shifter.

First to explicitly answer your question about realtime
scheduling. Yes I did put in the explicit enabling you
have told me, and yes, I get no error, i.e. realtime scheduling
should be set succesfully.

Roland, all of the stuff below looks good.

Do you have any other ideas what else I could try?
Sorry, this may have been in an earlier message, but what ethernet
chip/card are you using? (lspci will show the details.)
We’ve had lots of problems with RealTek.

What kind of CPU are you running on?

In my exerience, it takes at least a Core 2 Duo running at 3+ GHz to
transmit at full speed. The challenge is that there’s not much buffer
in the USRP2, and thus the average Tx rate over a very small window
needs to be rock solid. If the cpu gets side-tracked doing something
else, you’ll see underruns.

Also, on some machines with multiple CPU sockets, such as my dual quad
core Xeon, on some versions of the OS, it really matters as to whether
or not all of the computation is running on a single physical chip.
If you’ve got a machine like that, try using numactl to force all the
threads to live on a single physical chip (not core). If you look at
the output of cat /proc/cpuinfo, “physical id” indicates which cores
are in which sockets. (How the processors are assigned numbers seems
to have a random component (at least under some versions of the OS))

Turning to receive side: I invoked usrp2_fft.py -e eth2 -d 4
and I can see not see ‘S’ on the terminal. Am I correct in the
assumption this means I have no missed packets on receive?

Yes.

Thank you for your help!

You’re welcome.

Eric