Northern
Utah
WebSDR
|
Using "KA9Q-Radio"
as a multi-band front end for the PA3FWM WebSDR
|
What this page is about
The PA3FWM WebSDR (see the site "websdr.org" 3
) is widely used as a multi-user interface to virtual receivers allowing hundreds of people to use the same receive hardware simultaneously.
Typically, eight separate (hardware) receivers are required to fully populate the (maximum) eight bands that this can support - but using ka9q-radio and the RX888 Mk 2 simultaneous coverage of any eight bands between 2200 meters and 10 meters can be done with a single receiver. This page describes how that might be done.
Note: An update to this - using Fifos instead of ALSA - may be found on the page: High rate feeds for the PA3FWM WebSDR using FIFO sources from the RX-888 - link.
Important:
This document represents an effort on my part to understand the operation
of "ka9q-radio" and is not intended to be authoritative.
As such, this is a work in progress
and will certainly contain many "blank spots" and errors.
What it is intended to do is to help the new user along and
start to get the "feel" of how the pieces go together.
Please read EVERY
document in the /docs directory of the "ka9q-radio" git - and refer
back when you see something you don't understand!
Notice:
KA9Q-radio has undergone recent changes to improve
efficiency and performance. One aspect of this is that the
current version found at ka9q's github rolls the interfacing of the
receiver hardware directly into the main "radiod" processor rather than
having external, hardware-specific modules that interface the raw
samples via multicast.
What this means is that these pages currently refer to the old KA9Q-radio regime rather than the new integrated versions of radiod.
These pages will be updated with information related to the newer version as I have time to do so.
|
For more information about ka9q-radio, go here:
This page has much more information about the internal workings of ka9q-radio and other examples of its use.
First, what is "KA9Q-Radio" - and why is it different from other SDR
programs/suites?
One of the advantages of SDRs is the capability of
receiving multiple signals at the same time - but this is typically
exploited only in a limited fashion. The limit of this
capability is a combination of both the bandwidth of the acquisition
device (e.g. how much
spectrum the device is inhaling) and also the processing
capability of the host computer. Usually it's the latter
point that has limited the usefulness/capability of many wide-bandwidth
SDRs: It is typical for each "instance" of a receiver used by
a user to have to process data from the high-bandwidth acquisition
stream - which may be several megasamples. Because each
per-user instance requires so much processing, this can make a
multi-user receiver system "un-scalable" - that is, each user requires
a significant amount of CPU processing.
In 2006, an article was published 1
that described what might be considered to be a mathematical "shortcut"
when it comes to processing large amounts of data. Without
going
into detail, the "traditional" method for producing a single virtual
receivers is to crunch the full bandwidth data to yield - at least in
an amateur radio application - only a narrow bandwidth - perhaps a few
kHz for an SSB or FM signal or even a few 10s of kHz for a waterfall -
and if multiple receivers are required, it's necessary to "re-crunch"
the large amount of raw input data for each, individual receiver even
through that mathematical operation for each receiver is expensive in
CPU time and nearly identical. A far more efficient method -
potentially one that is many hundreds
of times more efficient, depending on how much "economy of scale" was
done - would be to do the "expensive" number crunching just once and
then use that already-processed data to synthesize each, individual
receiver - and it is this
method, generally referred to "Overlap and Save" - that is used by
KA9Q-Radio.
As an example of the "former" method: If the "csdr" 2
utility is used on, say, an RTL-SDR
with 2 MHz of bandwidth, a Raspberry Pi4 is capable of only handling
4-8 simultaneous receivers before all available CPU cycles are used:
This is one of the reasons why the open-source "OpenWebRX"
server isn't particularly salable to a large number (e.g. dozens) of
users. Conversely, the PA3FWM WebSDR server (which is closed source)
likely (unconfirmed!)
uses same the techniques as KA9Q-Radio - which are noted in Footnote #1 -to allow hundreds
of users on the same hardware platform as an OpenWebRX server that may
be only to handle a half-dozen or so.
Using the aforementioned "Overlap and Save" method in reference #1, a
Raspberry Pi4 running ka9q-radio can simultaneously decode every
FM channel in
the same 2 MHz bandwidth simultaneously
with plenty of processing power to spare!
KA9Q-radio is open-source and it may be found here: https://github.com/ka9q/ka9q-radio/
- and the instructions for compiling it along with a list of
dependencies may be found here: https://github.com/ka9q/ka9q-radio/blob/main/INSTALL.txt
Hardware requirements:
This article describes using the RX-888 (Mk2) SDR which
can operate at a sample rate exceeding 120 MSPS, implying a serial data
rate well above 3 Gbps. What this means is that for the
RX-888 a USB3
port is required with the RX-888 being connected directly to it rather
than through a USB hub. Additionally, mid-high range Intel
i5, i7 processor is necessary to operate the RX-888 at or above 60 MSPS
- the rate at which it can inhale the entire LF, MF and HF spectrum.
As noted elsewhere in this document, ka9q-radio will also work with
other popular SDR hardware: These other receivers typically
operate at lower bandwidths and may not require a USB3 interface or
as-powerful processing - but the discussion of the use of these other
devices is not covered here. (To Do: Another
document giving a brief overview using these other devices.)
Installing ka9q-radio
Information about the installation of ka9q-radio may be found on the page: ka9q_radio_installation
Using ka9q-radio with the RX-888 Mk2
There is also a page specific to using ka9q-radio with the RX-888 Mk2 SDR found here: Using the ka9q-radio utility with the RX-888
It is STRONGLY recommended that you review both of the above pages, get ka9q-radio and the RX-888 installed and working before proceeding.
Configuring for WebSDR use
Important note about the 768 kHz maximum bandwidth coverage - Using the ALSA versus "RTL_TCP" signal paths:
This page describes a method by which up to 768 kHzof
spectrum may be covered on a single WebSDR "band". This
maximum of 768 kHz is based on the hard-coded limits within the ALSA
sound system in Linux and exceeding this requires a modification of the
kernel. While the "rtl_tcp" signal path does not have a 768 kHz
limit, it is only 8 bits wide which means that it is significantly
limited in its capability of conveying bandwidths where there are
simultaneously both very weak and very strong signals.
The ka9q-radio software uses several configuration files that must be checked/modified.
Important - be aware of the following:
The configuration files discussed may be found in /etc/radio - but
you may also note that copies of these same files are in the home
directory (~/ka9q-radio). As of this writing there are a few caveats related to these files:
- When modules are run as a service, the configuration files in /etc/radio are used. It is these files that should be modified as needed.
- When modules run as a program there is currently inconsistency in how the various configuration files are used. For example:
- When running "rx888d" as either a program (e.g. "./rx888d g5rv &") or as a service (e.g. "sudo systemctl stop
rx888d@g5rv.service") it is always the file "/etc/radio/rx888d.conf" that will be used.
- When running "radiod" as a program (e.g. "./radiod radiod@hf.conf &") the file "~/ka9q-radio/radiod@hf.conf" will be used, but if you run it as a service, the configuration file "/etc/radio/radiod@hf.conf" may be used.
- When updating ka9q-radio from github, configuration files may be overwritten. If you are modifying/using an existing file, any changes you make will be lost when you update ka9q-radio.
If you are creating a new file, you are safe. In other
words, either back up and restore your files when you update, or use
your own, custom files with names other than the default files.
- For example, if you make changes to "rx888d.conf", these will be lost when you update ka9q-radio. If you use "rx888d" with the "-f" option, you may specify a different configuration file, and if this has your own, custom name, it should not be overwritten.
It is expected that these inconsistencies will be resolved in
future, but for now be prepared to back-up and restore your custom
configuration files and beware!
Starting "rx888d"
The "rx888d" program is that which interfaces with the RX-888 hardware,
placing the raw I/Q data on the network as multicast data. Even
running at half-bandwidth (e.g. 64.8 Msps rate)
the bit rate from the RX-888 on the USB interface will exceed the
capabilities of a Gig-E Ethernet port so data will usually be sent via
local loopback.
The invocation of rx888 is: ./rx888d g5rv &
This specifies the use of the [g5rv] section of the "rx888d.conf" file, described below.
The rx888d.conf file
The first of these is /etc/radio/rx888d.conf. The contents of this file - and the other configuration files - are discussed on the page "Configuration files in KA9Q-Radio" (It is recommended that you read this file, too!) in detail so we'll go over only basics here.
If you look in the "default" version of this file you will see:
[g5rv]
# ka9q customized
description = "G5RV RX888"
firmware = SDDC_FX3.img
samprate = 64800000 ; 2^8 * 3^4 * 5^5
# needs fftw3 wisdom and/or fft-threads >= 4 and some buffer tuning
# seems to lose data in the network path
# forward FFT for 129,600,000 Hz, 20ms and overlap = 5 is 3240000
#samprate = 129600000 ; 2^9 * 3^4 * 5^5
iface =
eth0
; force
primary interface, avoid wifi
status = rx888-status.local
data = rx888-pcm.local
ssrc = 10
;gain = 1.5 ; dB
gain = 10 ;dB - near floor of NF curve, still not too high for my G5RV
gainmode = high ; higher gain range
Again, refer to "Configuration files in KA9Q-Radio" for details, but the above file need not be changed for reception from 0-30 MHz as the sample rate (64.8 Msps) is adequate for our needs.
Comments:
- The RX-888 MK2 is capable of sample rates of up to around
130 Msps and thus it has a low-pass filter set to 55-60 MHz. What
this means is that at a lower sample rate than this - such as the 64.8
Msps above - there exists the possibility of aliasing of signals above the Nyquist frequency (e.g. half the sample rate, or 32.4 MHz).
- For
HF use with a sample rate in the 60-70 Msps range it is recommended
that a 30 MHz low-pass filter be used to avoid images/aliasing.
- This configuration was tested on a moderately-powerful
computer - an eight-core 3.4 GHz Intel i7-6700. While this
processor works well for eight virtual receivers of 768 kHz each and running the PA3FWM WebSDR software with the same configuration with the RX-888 running at 64.8 Msps, it does NOT quite this same receiver configuration with the RX-888 running at 129.6 Msps.
- If you make changes to "rx888d.conf", note the warning, above, about these modifications being overwritten when ka9q-radio is updated.
Invoking "radiod" and using the "radiod@websdr.conf" file
The "radiod" program is that which pulls the individual receivers'
spectral from the mass of raw receive data from the "rx888d". For
this we have a configuration file called "radiod@websdr.conf".
Here, we start "radiod" using the configuration file "radiod@websdr.conf":
./radiod radiod@websdr.conf &
The "radio@websdr.conf" file defines the "virtual" receivers that we need.
Here, we not only define the center frequency of each receiver,
but also the "mode" on which the receiver operates and the sample rate
at which this receiver operates. As we'll see, we can define our
own "modes" and bandwidths as our needs require. For example, for
some bands we'll need more bandwidth than others - 80 and 15 meters are
significantly larger than 30, 17 and 12 meters, for example - and we
can define these virtual receivers accordingly to provide reasonable
bandwidth to cover them.
Important note: Via the 16 bit ALSA interface the PA3FWM WebSDR is known to work properly ONLY at the following sample rates: 24, 48, 96, 192, 384 and 768 kHz. It may work at 1536 and higher, but the ALSA kernel would need to be modified to allow it.
Let us consider an example of a configuration file that may be used to cover every amateur band from 0 to 30 MHz:
# allowable Opus block times: 2.5, 5, 10, 20, 40, 60, 80, 100, 120
[global]
overlap = 5
blocktime = 50
input = rx888-status.local
samprate = 768000
mode = websdr768
status = hf.local
fft-threads = 4
[WEBSDR_768]
# Use 768 kHz BW to cover larger amateur bands
data = websdr768-pcm.local
mode = websdr768
freq = "3700k 5300k 7125k 14150k 21200k 28350k 28850k 29350k"
samprate = 768000
channels=2
low = -383000
high = 383000
agc = 0
gain = 0
[WEBSDR_384]
# 384 kHz to cover medium-sized amateur bands
data = websdr384-pcm.local
mode = websdr384
freq = "307k 1900k"
samprate = 384000
channels=2
low = -191500
high = 191500
agc = 0
gain = 0
[WEBSDR_192]
#192 kHz to cover the smaller amateur bands
data = websdr192-pcm.local
mode = websdr192
freq = "10090k 18118k 24940k"
samprate = 192000
channels=2
low = -95750
high = 95750
agc = 0
gain = 0
COMMENT:
- As discussed below, it may be necessary to apply different
"gain" values to different HF bands. For example, a higher "gain"
value may be appropriate for 15, 12 and 10 meters than for 160, 80 and
40 meters. As the gain values apply to all
frequencies defined in each section, you may need to split the sections
according to general frequency range, such as "[WEBSDR_768_LOW]" and
"[WEBSDR_768_HIGH]" for 160-40 and 15-10 meters, respectively.
- If you do this, be sure to change the name of the "data" parameter accordingly - and be absolutely sure to reflect that information in the "pcmdata" parameter of the various sections of the "pcmstart.sh" file described below.
For more information about the configuration files, see "Configuration files in KA9Q-Radio".
In the [global] section of the file we see more or less what was copied from "radiod@hf.conf" - our template - with a few differences:
- samprate = 768000 -
This sets the default sample rate for the sections below at 768 kHz.
Its use is optional if you specifically state the sample rate in
the sections that follow.
- mode = websdr768 -
This sets the default "mode" for the sections below as our
custom-defined "websdr" mode. Its use is optional if you
specifically state the mode in the sections that follow.
In the sections that follow we define virtual receivers with various
bandwidth commensurate to the amateur bands that they cover.
Because we can arbitrarily define receivers and their bandwidths,
we can select the sample rate of the virtual receiver to better-suit
the band being covered. For example, using 192 kHz for 30 meters
makes sense as it is only 50 kHz wide, anyway - and running some bands
at lower sample rates can reduce CPU loading in both radiod and the WebSDR.
In all sections, we have a few important definitions:
- low and high - These define the lower and upper edges of the receive bandwidth. For ka9q-radio these can be about as wide as half the sample rate.
- agc = 0 - This
disables AGC on the virtual receiver. We don't need AGC here,
anyway, as the WebSDR has it already and since the RX-888 is a
direct-sampling SDR, the only place AGC would make sense would be its
RF input.
- gain = 0 - This sets
the gain to zero meaning that the amplitude is at approximately the
same scale as the RF signal. Please read the discussion of this value in a section below.
While the bandwidth, AGC and gain are actually defined in the "modes" file (described below) but it is a good idea to explicity define them here (again) should other defaults be specified in the radiod configuration file.
In the [WEBSDR_768] section we
define bands that are large enough to require such bandwidth - namely
80, 20 and 15 meters. 40 meters is included in this section as
there are signals of interest adjacent to this band.
- data = websdr768-pcm.local - This defines that our receive multicast streams will have the name "websdr768-pcm.local".
- mode = websdr768 - This specifies our custom-defined receive mode called "websdr768" for 768 kHz bandwidth which we will discuss shortly. Since
we specified this in the [global] section, strictly speaking, we don't
need to specify it here - although doing so improves clarity.
- freq = "....."
- This line specifies the center frequencies of all of our virtual
receivers. If you wish, you can add/remove frequencies from this
line to create more virtual receivers or save CPU power.
- With 10 meters being 1.7 MHz wide, it cannot be covered
with a single 768 kHz wide receiver so you may note that we have
defined three
10 meter frequencies that overlap each other when a 768 kHz sample rate
is used. As mentioned elsewhere, because Multicast is used to
disseminate receiver data, you can feed as many other devices the same
data (as long as they are on your LAN) as you wish - and this could include more than one WebSDR server - so our definition of multiple frequencies
- samplerate = 768000 - This defines that we are using 768 kHz for each receiver defined in this section. For some bands (e.g. 160, 30, 17, 12 meters) this amount of bandwidth is overkill so we'll define those "smaller" bands in different sections, below.
- Since we specified this sample rate as the default in the
[global] section, strictly speaking, we don't need to specify it here -
although doing so improves clarity.
- channels = 2 - We need to convey I/Q data to the WebSDR, so we need two-channel data.
- low = -383000 and high = 383000
- Define the filtered bandwidth. Because of the method used, we
can specify 1 kHz short of the theoretical maximum +/-384 kHz. (Do not specify exactly 384 kHz or else objectionable artifacts can occur if signals are right on these edges.)
In the [WEBSDR_384]
section we define bands more appropriate to 384 kHz of coverage -
namely a receiver centered at 307 kHz to cover both the 2200 and 630
meter bands and the U.S. 160 meter band which is 200 kHz wide.
The parameters that are modified are:
- data = websdr384-pcm.local - We define a different stream for the 384 kHz bands.
- freq = "....." - Here we define the center frequencies of the bands.
- mode = websdr384 - This specifies our custom-defined receive mode called "websdr384" for 384 kHz bandwidth which we will discuss shortly.
- samplerate = 384000 - Define the use of 384 kHz for these bands as this is a better "fit" for the amount of spectrum we need.
- low = -191500 and high = 191500
- Define the filtered bandwidth. Because of the method used, we can
specify 500 Hz short of the theoretical maximum +/-192 kHz. (Do not specify exactly 192 kHz or else objectionable artifacts can occur if signals are right on these edges.)
In the [WEBSDR_192]
section we define those bands for which 192 kHz coverage is appropriate
- namely the 30 meter band and the 17 and 12 meter bands. The
parameters that are modified are:
- data = websdr192-pcm.local - We define a stream for the 192 kHz bands.
- freq = "....." - Here we define the center frequencies of the bands.
- mode = websdr192 - This specifies our custom-defined receive mode called "websdr192" for 192 kHz bandwidth which we will discuss shortly.
- samplerate = 192000 - Define the use of 192 kHz for these bands.
- low = -95750 and high = 95750
- Define the filtered bandwidth. Because of the method used, we can
specify 250Hz short of the theoretical maximum +/-96 kHz. (Do not specify exactly 96 kHz or else objectionable artifacts can occur if signals are right on these edges.)
As mentioned earlier, the PA3FWM can (currently)
support only eight bands simultaneously, so you might think that having
more than eight frequencies is superfluous - but this is not
necessarily so:
- If you have a single WebSDR server:
You might want to edit the "freq" lines to select just the eight
bands that you wish to cover as this will save some CPU processing
power.
- If you have more than one WebSDR server: You can use a single instance of ka9q-radio to service multiple WebSDRs due to the fact that the received data is conveyed via multicast. Consider the following examples:
- Dedicated "radio server: You could have a server that is dedicated only to running ka9q-radio and send the receive data of the virtual receivers, via multicast, to one or other WebSDR servers on the same LAN.
- If you have multiple WebSDR servers, you can cover more than eight bands simultaneously. For instance, if you have two
WebSDR servers, you could cover all of the frequencies mentioned above
and then, if you wished, you could define even more bands such as those
for Shortwave Broadcast reception.
- Combined WebSDR and "radio" server: If the host computer has the necessary processing power, you can run the WebSDR software on the same computer as being used for the receiver hardware. Here are some possible configurations:
- One server for the receiver hardware and one instance of
WebSDR. Even in this case if you wish, you could have additional
multicast streams propagate across the LAN to feed receiver data to
another server (or two) running WebSDR software.
- One server for the receiver hardware and multiple
"virtual" machines running additional instance of WebSDR. If your
host computer is powerful enough you could actually run more than one
instance of the WebSDR software - each in its own "virtual" machine.
- Non-WebSDR feeds: If you wish, you can either use the existing virtual receivers described above or create new ones to produce additional
multicast streams to provide sources to things like a CW skimmer, WSPR
monitoring, FT-8 decoding, etc. - all while feeding data to the WebSDR
Server(s).
Adding websdr "modes" to "modes.conf"
Referenced above is a "mode" called "websdr". The ka9q-radio package has the ability to arbitrarily define a mode in terms of type ("linear" as in am/cw/usb, etc. and not linear as in fm), sample rate, bandwidth, and several other parameters: This is discussed in more detail in "Configuration files in KA9Q-Radio". (Note: See the warning at the top of the page about .conf files being overwritten when updates occur).
For our purposes we need to add some new "modes" for WebSDR use and
this is done in the "modes.conf" file. Specifically, we need to
supply I/Q data to the WebSDR and we would like to specify both the
bandwidth and sample rate.
NOTE: While ka9q-radio
previously passed the sample rate and filter parameters so that they
would not need to explicitly defined in the "radiod" configuration
file, this may not be the case - particularly if you have defined
defaults in that file. Be prepared to explicity specify the
sample rate, high/low filter edges, agc, gain and number of channels in
the "radiod" configuration file.
Somewhere in the "modes.conf" file we add the following lines to define
our new modes, having previously started with the already-defined
"[iq]" mode as a template:
[websdr768]
demod = linear
samprate = 768000
low = -383000
high = 383000
channels = 2
agc = 0
gain = 0
[websdr384]
demod = linear
samprate = 384000
low = -191500
high = 191500
channels = 2
agc = 0
gain = 0
[websdr192]
demod = linear
samprate = 192000
low = -95750
high = 95750
channels = 2
agc = 0
gain = 0
These three sections define the sample rates that we need (768, 384 and 192 kHz) and their filtered bandwidths:
- [websdrxxx] - This is the name of our new "mode" using a number for "xxx" to indicate the sample rate (e.g. [websdr768]).
- demod = linear - We are using a "linear" mode (e.g. NOT FM)
- samprate = xxxx - This defines the sample rate of the "mode" that we are defining
- low and high - These define the lower and upper edges of the receive bandwidth. For ka9q-radio these can be about as wide as half the sample rate.
- channels = 2 - Because we are sending I/Q data, we use two channels.
- agc = 0 - This
disables AGC on the virtual receiver. We don't need AGC here,
anyway, as the WebSDR has it already and since the RX-888 is a
direct-sampling SDR, the only place AGC would make sense would be its
RF input.
- gain = 0 - This sets
the gain to zero meaning that the amplitude is at approximately the
same scale as the RF signal. Please read the discussion of this value in a section below.
Discussion:
Above we see our three websdr "modes" - which differ only
by the sample rate and the covered bandwidth. It's worth noting
that we can define arbitrary receiver bandwidths if we like using the
"low" and "high" parameters to set the low and high edge with respect
to the center frequency of our receiver. In the case of the
[websdr384] mode could theoretically define our edges as being -192000
and 192000 meaning that we have a total
of 384 kHz - which is exactly as much as will fit within our 384 kHz
sample rate using I/Q data. In practice, we wouldn't want to do
this. As it happens, placing a signal exactly at the edge of either the upper or lower limit of the filter (e.g. right at + or - 192 kHz in this case)
causes a odd noise effects that can cause significant disruption within
several percent of passband but if we narrow the filter very slightly -
say to +/- 190 kHz - this effect is very significantly reduced.
It might raise eyebrows that our defined filter bandwidth (+/- 192 kHz) is
so close to that permitted by the sample rate in for the reasons that
that no filter is perfect in that it might be able to cut off signals
beyond this adequately and aliases in the form of signals just beyond
the edge of the filter will appear just inside the other end of the
spectrum. For those who have used sound card based
receivers this may be a familiar problem: A signal that is just
above the upper end of the receiver will actually appear in weaker form
at the bottom as an aliased response - the amount of diminution being
related to how well the sound card's hardware filtering actually works.
In the case of ka9q-radio since
we are synthesizing our virtual receiver bandwidths from a much larger
pool of FFT data we have the advantage of more-easily being able to
define our bandwidth in discrete segments of bandwidth within the FFT (called "bins") and when we need a given bandwidth we can, by knowing which bins contain the portion of spectrum that we need, select only those bins. What this means is that we have a very close
to a "brick wall" type of response and that we can, indeed, specify our
receiver as being just 380 kHz wide at a 384 kHz sample rate! In
reality there is a tiny
bit of aliasing at the edges owing to the fact that a signal in one of
the FFT bins bleed slightly into their neighbors, but this effect is
very minor and since the edge of a filter is typically placed outside
an amateur band, it is unlikely to ever be noticed.
Get it up and running
At this point the reader should take a look at the following pages:
- Operating a WebSDR receiver at 384 or 768 kHz using the 16 bit signal path - This
provides a brief overview of how one might go about increasing the
coverage of a signal "band" on a WebSDR server to 384 or 768 kHz on the
16 bit signal path.
- Configuring high-rate audio loopback devices - One of the features of Linux is the ability to configure virtual sound cards. Doing so adds flexibility when sending audio (including raw I/Q streams from the receiver)
from a particular device driver into another program, whether it be a
WebSDR server and/or another application that might use this same
receive stream. This page describes one way of setting up such
loopback devices, and how to build a patched version of the loopback
module that is capable of operating up through at least 768 kHz.
- Creating "fplay", a version of the Linux "aplay" utility, but without the speed limit - While more recent versions of Ubuntu (and likely other)
Linux distributions have ALSA kernels capable of sample rates of at
least 768 ksps, many of the related utilities do not - including
"aplay". This page describes how to create "fplay", a version of
"aplay" that has no such sample rate speed limit - and why this utility
is helpful when implementing a "high rate" 16 bit ALSA device like the
SDRPlay RSP1a.
- Getting I/Q data from the receiver - This
page discusses how raw I/Q data may be routed from the receiver and
into ALSA. Specifically, it discusses the "sdrplayalsa" utility -
code written specifically for the RPS1a, and how/why it's built-in AGC
may be used.
- Duplicating receiver I/Q data streams using ALSA and asoundrc - Normally, the raw I/Q data from the receiver on a WebSDR goes just ONE
place, but wouldn't it be nice if that data could be replicated and
used elsewhere for things like analysis, reception of digital signals
like FT-8 and WSPR, a CW skimmer, logging or monitoring the health of
the receiver system itself? This page describes how it's possible
to get the raw I/Q data that you already have and duplicate it, making it available for other purposes to leverage the capabilities of your system?
These pages describe the specific software configurations (and utilities) required for our task and from here on, it is presumed that these have been done and that they are working.
The "pcmstart.sh" file:
An additional script is required to get data from ka9q-radio to the WebSDR, via the loopback mechanism described above. For this task we are going to use pcmcat - one of the programs that comes with ka9q-radio - to take the multicast stream from a virtual receiver and pass it into the ALSA system where it can be used by the WebSDR.
The contents of a sample "pcmstart.sh" file are as follows:
#! /bin/bash
#
# NOTICE: Be sure that the ssrc values in this file MATCH the frequencies specified in your "radiod" configuration file!
# If the do NOT match, you will get no signals the bands where the ssrc and frequency do not match.
#kill any already-running instances
killall pcmcat
killall fplay
sleep 1
buffer=75000
logfile="logrx"
# A receiver's log file is called "logrx#" where "#" is the loopback channel # (e.g. "logrx0", "logrx1", etc.)
loopch=0
ssrc=1900
sr=384000
pcmdata="websdr384-pcm.local"
pcmcat $pcmdata -2 -s $ssrc | ./fplay -D plug:rx_ch$loopch -r $sr -c 2
-f s16_le --disable-softvol -B $buffer & # < /dev/null 2>>
$logfile\rx$loopch &
loopch=1
ssrc=3700
sr=768000
pcmdata="websdr768-pcm.local"
pcmcat $pcmdata -2 -s $ssrc | ./fplay -D plug:rx_ch$loopch -r $sr -c 2 -f s16_le --disable-softvol -B $buffer & # < /dev/null 2>> $logfile\rx$loopch &
loopch=2
ssrc=7125
sr=768000
pcmdata="websdr768-pcm.local"
pcmcat $pcmdata -2 -s $ssrc | ./fplay -D plug:rx_ch$loopch -r $sr -c 2 -f s16_le --disable-softvol -B $buffer & # < /dev/null 2>> $logfile\rx$loopch &
loopch=3
ssrc=14150
sr=768000
pcmdata="websdr768-pcm.local"
pcmcat $pcmdata -2 -s $ssrc | ./fplay -D plug:rx_ch$loopch -r $sr -c 2 -f s16_le --disable-softvol -B $buffer & # < /dev/null 2>> $logfile\rx$loopch &
loopch=4
ssrc=18118
sr=192000
pcmdata="websdr192-pcm.local"
pcmcat $pcmdata -2 -s $ssrc | ./fplay -D plug:rx_ch$loopch -r $sr -c 2 -f s16_le --disable-softvol -B $buffer & # < /dev/null 2>> $logfile\rx$loopch &
loopch=5
ssrc=21200
sr=768000
pcmdata="websdr768-pcm.local"
pcmcat $pcmdata -2 -s $ssrc | ./fplay -D plug:rx_ch$loopch -r $sr -c 2 -f s16_le --disable-softvol -B $buffer & # < /dev/null 2>> $logfile\rx$loopch &
loopch=6
ssrc=24940
sr=192000
pcmdata="websdr192-pcm.local"
pcmcat $pcmdata -2 -s $ssrc | ./fplay -D plug:rx_ch$loopch -r $sr -c 2 -f s16_le --disable-softvol -B $buffer & # < /dev/null 2>> $logfile\rx$loopch &
loopch=7
ssrc=28350
sr=768000
pcmdata="websdr768-pcm.local"
pcmcat $pcmdata -2 -s $ssrc | ./fplay -D plug:rx_ch$loopch -r $sr -c 2 -f s16_le --disable-softvol -B $buffer & # < /dev/null 2>> $logfile\rx$loopch &
|
Important: This script should be executed before starting the WebSDR server, but after ka9q-radio is running.
Dependencies:
- This script should be marked as executable via "chmod".
- The above script requires that the "fplay" binary be located
in the same directory as this script as indicated by the "./fplay"
invocations.
- The above script also presumes that you have "fastloop" running as described in the "high rate loopback" article linked above.
- The file ".asoundrc" needs to be installed in the home directory of the user running ka9q-radio to configure the loopback devices and given appropriate permission. This may be found here.
The configurations in detail:
- As we can see in the above file, there are eight instances - each one given a number from 0-7 in the "loopch" variable: This describes which loopback device (defined in ".asoundrc")
- The variable "sr" specifies the sample rate for each, individual receiver as defined in the "radiod@websdr.conf" file described earlier.
- The variable "pcmdata" specifies the name of the multicast stream on which the virtual receiver's data can be found.
- This is the name in the data field in the relevant section in the "radiod@websdr.conf" file.
- ssrc is the "Stream Source Identifier" of the specific virtual receiver. In the case of the configurations in "radiod@websdr.conf" this is the frequency of the virtual receiver with any non-numerals removed.
- Examples:
- For 160 meters, the frequency specified in "radiod@websdr.conf" is "1900k" so the ssrc would be "1900"
- For 15 meters, the frequency is specified in "radiod@websdr.conf" is "21200k" so the ssrc would be "21200"
- Note: There are multiple ways that a frequency can be specified and this can affect the resulting SSRC - see "Configuration files in KA9Q-Radio" for more information about this.
- IMPORTANT: If you change the frequency of the virtual receiver, you will also change its SSRC:
- If the SSRC changes because you changed the frequency
- You must change it in this file
- You must also change the frequency in "websdr.cfg" (the main WebSDR configuration file) to match and restart the WebSDR.
- The data being emitted by "pcmcat" is taken by "fplay" - a
modified version of "aplay" and sent to the input of the loopback
device (e.g. virtual sound card) with the "-D plug:rx_chX" where "X" is the number of the loopback device. The sample rate ("-r $sr") is explicitly specified, as is the use of 2-channels and the use of standard 16 bit data (the "-c 2" and "-f s16_le" parameters). Finally, a 50000 microsecond (0.05 second)
buffer is specified with the "-B" parameter to prevent buffer underruns
as this would otherwise be likely to occur on a Linux system where many
tasks are being shared/scheduled.
- Note: For
signal paths running at 192 kHz or lower sample rate, the "aplay"
utility could be used, as could the "snd-aloop" rather than the
modified "fplay" and "fastloop" utilities mentioned.
How this script works:
Included with ka9q-radio
is a program called "pcmcat" - the name being related to the "cat"
utility in Linux - that will spit out the data on the specified stream
to STDOUT. In the cases above, we are using data from the streams
defined in "radiod@websdr.conf" namely "websdr768-pcm.local",
"websdr384-pcm.local" and "websdr192-pcm.local" and we are further
specifying the sub
streams using the "ssrc" which happen to be designated by the tuned
frequency. Using the Linux "pipe" (the "|" character) we are
feeding the data from pcmcat into the "fplay" utility (a version of "aplay" modified to be able to operate at higher than 192 kHz) to put this data into an ALSA loopback device (e.g. a "virtual" sound card) defined in the "asoundrc" file.
With the I/Q data now in a virtual ALSA device, it is available to the WebSDR as just another sound device (e.g. of type "plug") as described below.
Using the loopback in the "websdr.cfg" file
The PA3FWM WebSDR uses a file called "websdr.cfg" (typically located in ~/cfg)
for its band/frequency configurations. Based on our
configuration, above, let's discuss some examples of the portion of
that file that configures the bands.
Here is a definition for 80 meters based on what we have already described:
band 80
# loopbacks are used with "f_loopX_out" where "X" is the #band number, 0-7
device $plug:f_loop1_out0
centerfreq 3700
gain -20
samplerate 768000
# We get "swapped" I/Q data from ka9q-radio
swapiq
# set "hpf" to zero to prevent "zero-Hz hole" in the middle of the receiver
hpf 0
noiseblanker 30
antenna wet noodle
extrazoom 1
In detail:
- band 80 - The text following "band" is what will appear on the WebSDR control screen - in this case, it's just "80" for 80 meters.
- device $plug:f_loop1_out0
- In our "pcmstart.sh" file we define "loopch = 1" meaning that our 80
meter data is on loopback #1 - and the "loop1" portion of our audio
device specifies this. This device is defined in ".asoundrc".
- Note: The loopback itself can send data to more than
one place. For example, if we needed 768 kHz 80 meter data
somewhere else, we could get it from a device called
"$plug:f_loop1_out1" where "out1" is another output spigot of this
data. With the configuration described in the ".asoundrc" file
and the "high rate loopback devices" article we can supply up to eight such duplications of data.
- centerfreq 3700 - This is the center frequency of our receiver as defined in our "radiod@websdr.conf" file. The frequency here must match our configured center frequency if you want the tuning to be correct.
- gain -20 - Here we
set the gain to "-20". In the WebSDR software, the "gain" is a
bit arbitrary and it is typically set such that the waterfall is
"dark-ish" on a dead band with no signals - but there's an issue with
this:
- On lower bands, the noise floor is rather high, so a higher value of "gain" (e.g. less-negative)
is typically used. On higher bands the noise floor is naturally
much lower so the same setting that yielded a dark-ish waterfall on,
say, 80 meters, would show a black screen on 10 meters on a "dead"
band. The obvious "fix" to this is to use a higher (less-negative) "gain" number for the higher bands.
- The problem with this is that the "gain" value also
affects the on-screen S-meter. For the lower bands, if you
calibrate the S-meter to a known-accurate signal generator the
waterfall brightness may be acceptable - but doing this on a higher
band will result in a (more or less) black waterfall.
- If you want to have "non-black" waterfalls on dead bands and an S-meter that accurately reports the signals you'll need to slightly modify the javascript with the WebSDR - contact the author using the information below for details.
- swapiq - This tells the WebSDR that I/Q data should be swapped - something that's needed when we are using ka9q-radio as well as other receive devices.
- hpf 0 - WebSDR has a
high-pass filter enabled by default that helps remove the DC offset and
"near DC" noise present on many A/D converters. This phenomenon
is absent in ka9q-radio
in the configuration described above so we are disabling the filter or
else we would end up with a small "hole" at exactly the center
frequency.
- noiseblanker 30 - This helps reduce "clicks", power line noise and other noise that might appear on RF and its use is optional.
- antenna wet noodle -
This defines our antenna name as "wet noodle" for this particular band.
You would put in this a description of the type of antenna you
are actually using. If you are using the same antenna for all
bands - as the case is here - you want to make sure that this text is
exactly the same for every band.
- extrazoom 1 - With
our bandwidth being 768 kHz, we need to zoom in quite a bit to see
details of an SSB or CW signal so setting "extrazoom 1" allows one more
level of zooming-in than default but note that this tends to slow the waterfall a bit as
well as increasing CPU loading. Its use is optional.
Antenna/signal level considerations
The use of individual receivers affords the installation of band-specific filters to prevent very strong out-of-band energy (e.g. SWBC -strong shortwave broadcast stations, local mediumwave/AM stations)
from overloading the receiver. With a receiver like the RX-888,
this situation is more complicated: Even with 16 bits, this
is not enough bit depth/dynamic range to handle the entire range of
expected signals under a wide variety of HF propagation conditions.
As-is, the gain of the RX-888 Mk2 is slightly too low to receive the 10
meter noise floor on a theoretical "flat" antenna system in a very
quiet, rural area and it needs a bit of amplification. If you do
that, however, you will risk overload from strong SWBC signals and -
particularly in the summer - the very high static levels present on the
lower bands (160-40 meters). If you are using the RX-888 for receiving all HF bands, you ARE risking overload at times, but there are ways to mitigate this:
- Set the gain/amplification only as high as absolute necessary.
Typically, the worst-case is 10 meters where the natural
background noise is low, so your total system gain should be set just high enough so that you can hear no more than 6 dB of noise floor increase when your antenna is connected.
- Add a pre-emphasis filter.
As mentioned earlier, the natural noise floor on, say, 80 meters
is much higher than on 10 meters - around 20 dB higher and this means
is that for the lower bands, you will likely have excess gain. In that case you can significantly attenuate signals on the lower bands without losing any actual signal-noise ratio/sensitivity.
Because the analog-digital converter can only accept only so much total
RF energy before it overloads, reducing the total amount of RF energy
by reducing what is just going to be noise from the lower frequencies
will better-utilize that limited capability.
- Add an AM broadcast blocking filter. As mentioned above, there is a limit to the total
amount of energy that can impinge on the analog-digital converter in
the receiver - even if one isn't tuned there. A strong, local AM
broadcast band signal can contribute to this total energy and reduce
that which is available elsewhere so it would be prudent to add one of
the many-available blocking filters to your receiver.
- Add "window" filtering.
In the case of the WebSDR, it is unlikely that you will need to
listen to signals "between bands" as these may well not be covered by
the limited bandwidth of your eight-available bands. One such
option is a "window" filter - that is, a filter that will pass only
certain segments of the HF band but block others. Unfortunately,
such an item is not commercially available (as far as I know) but if you are able to build something, you can contact me at the information at the bottom of this page.
- Add a low-pass filter.
As mentioned near the top of this page, the RX-888 Mk2 can
operate at a sample rate of about 130 Msps - if you have a fairly
powerful computer. For HF-only use, running it at 64.8 Msps, as
shown in the example above, is perfectly adequate, but local signal
sources above 30 MHz may appear as aliases in the receive spectrum.
In this case, a 30 MHz low-pass filter - such as that used for HF
160-10 meter transmit use - is strongly recommended.
Defining the "gain" setting in the "modes.conf" and in the "radiod" configuration files
Previously we defined the "AGC" as being off (a value of zero)
and a gain value of zero. For the WebSDR we would prefer that the
AGC be turned off as this would defeat the utility of the "S" meter:
If the gain of the signal path varied due to AGC action, it would
not be possible to properly calibrate the S meter - at least not
without the WebSDR "knowing" the current gain setting. (Comment: This is possible to do, but currently not implemented.)
As for the "gain" value, a gain setting of zero means that the signal level being output by ka9q-radio is the same, in amplitude (represented by a 16 bit signed number)
as the signal at that frequency being applied to the antenna port.
When connected to an antenna, however, there will surely be many
other signals, meaning that it is unlikely that the 16 bit numerical
value seen on the WebSDR's signal path for that band will have any
relationship at all the raw 16 bit A/D converter value - and (almost) no matter what, that latter value will always be lower - probably MUCH lower!
This property arises when you consider that since the signal path
feeding the WebSDR will have a narrow bandwidth compared to the
acquisition bandwidth of the RX-888 (e.g. at best we will see 768 kHz of the 64.8 MHz sample rate)
this means that the signal output from the virtual receivers can be
extremely low - particularly on a band that is very quiet, or even
"closed". What this may mean is that on a higher band - say 12 or
10 meters - when it is closed and there are no local signals, you may
get 16 bit values feeding the WebSDR's signal input for that band being
represened by very few bits.
Fortunately, the PA3FWM WebSDR code has some diagnostic tools for this - specifically the "~~status" web page (this is detailed in the documentation supplied with the WebSDR code). Take this example:
name overruns readb. blocks timestamp users 10sec_ADmin/max ever_ADmin/max source
80 1 8 119949 1698981790.889891 0 -378 369 -1441 1440 0,$plug:f_loop0_out0
60 1 8 55707 1698975063.039457 0 -233 213 -921 936 0,$plug:f_loop1_out0
40M 1 8 11405 1698973169.597555 0 -236 210 -1080 1050 0,$plug:f_loop2_out0
20M 1 8 213185 1698977049.885691 0 -100 109 -322 318 0,$plug:f_loop3_out0
17M 0 2 2450 1698972784.537468 0 -47 44 -220 231 0,$plug:f_loop4_out0
15M 1 8 3904 1698973131.501877 0 -36 35 -141 153 0,$plug:f_loop5_out0
12M 0 2 2342 1698973129.014715 0 -13 15 -80 76 0,$plug:f_loop6_out0
10M 1 8 1741 1698973140.366123 0 -13 14 -82 82 0,$plug:f_loop7_out0
In looking at the above values, we see something that is
potentially alarming: Knowing that these signals are on a 16 bit
signal path, we can see that the 12 and 10 meter signals are generally
representable by just 4 bits meaning that it is likely that
quantization artifacts (intermodulation products, noise, etc.) will likely be present. As a starting point, the signal level should be in the 100-300 range with no signals present (on a "dead" band)
due to noise. What this means is that one will likely need to
tweak the gain settings over time to accommodate the signal levels
ranging from strong summer static on the low bands to very low winter
noise across all bands when they are "closed."
Prudent application of the "gain" setting in the configuration can help
remedy this situation. This value will scale the signals, in dB,
by the value specified and effectively "amplifying" the result:
As the internal signal processing within ka9q-radio
is done with greater precision than 16 bit signed arithmetic, this
amplification will yield better fidelity of the outputted signal to the
WebSDR (which is limited to 16 bit signed data) by multiplying it before it is converted to an integer value.
Setting the gain of the RX-888 front end and the signal path
The importance of proper gain and signal levels feeding the
RX-888 cannot be over-emphasized if good performance is to be obtained.
Consider the following two statements:
- The "attenuator" setting of the RX-888 (Mk2) should always be set to zero. There are no common scenarios where the attenuator built into the RX-888 (Mk2) is likely to be useful! The ability of the PGA (Programmable Gain Amplifier) within the RX-888 (Mk2) is sufficient to handle any signal levels that one is ever likely to see on an HF antenna.
- The most useful setting of the RX-888 (Mk2) gain is "20dB". A setting of 20 dB (which requires that the '888 gain be in the "high gain" mode, if that is a separate option) offers the best compromise between sensitivity and signal-handling characteristics.
The RX-888 - like ALL
other software-defined radios - use an analog-to-digital converter to
acquire the raw signal data into the digital realm. ALL analog-to-digital converters have a very defined limit of how much TOTAL signal power can be applied before it overloads, regardless of frequency. In other words, you MUST keep this total signal power safely below the overload limit (e.g. "clipping" threshold) or else signals all across the spectrum will be disrupted. It is imperative that one does their best to properly set the input signal level to achieve what are two conflicting requirements:
- The TOTAL
amount of signal must not exceed the "clipping" level of the
analog-to-digital converter: Higher than this causes degradation.
- The total gain of the receiver must be high enough so that on the bands with the lowest amounts of signal level (and likely noise)
the weakest signals and the noise spectra in which they reside are
represented by enough of the analog-to-digital converter's dynamic
range to be represented with adequate fidelity.
In other words, we have a problem: Too little gain, our
receiver is deaf - particularly on the high bands and too much gain, it
overloads and everything is disrupted.By
itself, the absolute sensitivity of the RX-888 (Mk2) is on the order of
-152dBm/Hz. For an urban installation where the noise level is
fairly high, this may be adequate, but if you are lucky enough to have
a very quiet RF environment - one that is near the ITU noise floor -
you will find that the RX-888 (Mk2)'s sensitivity is about 15 dB lower
than necessary to be able to hear weak signals on a quiet 10 meter band
- and the situation. What this means is that if you are planning
to use the RX-888 (Mk2) in a very quiet RF environment, you will need
15-20 dB of amplification - and this must take into account losses of
filters, splitters and feedline.
This issue was discussed in the following blog entry: Measuring the signal dynamics of the RX-888 (Mk2) - link.
There is another problem that arises: The noise floor of the HF spectrum is NOT flat: Noise at 80 meters is on the order of 20dB higher than that at 10 meters which means that if you do add amplification, the RX-888 (Mk2) will very likely overload on signals in the low HF spectrum (1-10 MHz) alone: This fact is true for ANY
direct-sampling HF receiver that is intended to simultaneously receive
everything up to 30 MHz - including the RX-888, KiwiSDR, Red Pitaya,
and any other!
This issue was discussed in the following blog entry: Revisiting the "Limited Attenuation High Pass" filter - again - link. This article describes a sort of pre-emphasis (or "shelving") filter that significantly attenuates lower-frequency signals (<10 MHz)
but progressively offers less attenuation by the time one gets to 30
MHz. As the signal and noise levels at the lower end of HF are so
high, the effect is not
to remove low frequency signals, but to generally "equalize" signals
across the HF spectrum so that low frequency and high frequency signals
- and the corresponding noise floor - is more level across the
frequency range. In other words: It minimizes the wasting
of valuable A/D dynamic range on lower frequencies where the total
noise power is very high and allows the insertion of the amplification
that is needed to allow reception near 30 MHz while reducing the
probability of overload.
In short: To get decent performance from a full-bandwidth, you should:
- Install filtering to attenuate the AM (Mediumwave) band.
If you have local AM broadcast stations, they can be quite strong
and can consume a considerable portion of the total RF power that the
receiver can handle. Filters that offer considerable attenuation
of this band are available inexpensively, but they typically impact 160
meters.
- If you wish to prevent signals on the AM broadcast band
from overloading the receiver, but still wish to be able to hear them,
it is possible to selectively attenuate them as described in this
article - Link.
- Use a pre-emphasis filter as described in the article linked above. This will reduce signal levels at low HF and prevent their total signal power from overloading the A/D converter.
Tools to do this:
- To manage the signal level into the front end of the RX-888, the ka9q-radio
"control" program will show front-end signal levels. This can be
used to determine if the signal level applied to the RX-888 is too high
or too low: If the latter, it may be necessary to add external, low-noise amplification and adjust the gain accordingly.
- Use the "~~status" URL on the PA3FWM WebSDR. This will
tell you if the signal level feeding the specific band of the WebSDR is
too-near the quantization limit and if adjustment of the "gain" setting
in the "radiod" configuration file is warranted.
- As noted above, it may be necessary to split the various
HF bands up into separate sections within the "radiod" configuration
file so that different gain values can be applied according to need.
Conclusion:
As described above, it is certainly practical to use a single
piece of receive hardware - the RX-888 Mk 2 - as the sole receiver for
a PA3FWM receiver system. The above configurations have actually
been tested on the workbench and found to perform as expected.
(And no, it's not lost on my
the irony that the techniques used by ka9q-radio are probably the same
used by the PA3FWM software to efficiently provide the large number of
individual virtual receivers that it does!)
For more information about ka9q-radio, go here:
This page has much more information about the internal workings of ka9q-radio and other examples of its use.
References:
1 - Mark Borgerding,
“Turning Overlap-Save into a Multiband Mixing, Downsampling
Filter Bank”, IEEE Signal Processing
Magazine, March 2006. https://www.iro.umontreal.ca/~mignotte/IFT3205/Documents/TipsAndTricks/MultibandFilterbank.pdf
2 - The csdr tools by HA7ILM may
be found here: https://github.com/ha7ilm/csdr.
This represents a "toolbox" of signal processing engines that
can do things like filter, decimate, shift, demodulate, convert
formats, provide AGC and more. These tools may be useful for
additional filtering of signals.
3 - The PA3FWM at the
University of Twente in the Netherlands uses an A/D converter that
streams raw data via an Ethernet interface to a computer that uses
graphics-card processors to do the heavy-lifting. See these
pages for more information: http://websdr.ewi.utwente.nl:8901/
and http://www.pa3fwm.nl/projects/sdr
P.T. de Boer, PA3FWM, provides a version of the WebSDR software that is
similar to that operating at the University of Twente, but does not
utilize GPU cores and bespoke hardware and is therefore more limited in
bandwidth - but it still is extremely
economical with its CPU power in servicing many users - even on
computers of limited processor power allowing many times the number of
users compared to OpenWebRX. While I have no direct evidence
that it does, I suspect
that the PA3FWM WebSDR uses the technique described in Reference #1 to
allow its very efficient use of CPU power to service many users
simultaneously.
TO DO:
- Find
the best way to determine multicast addresses for PCM and status streams
- DONE - use AVAHI
- Do a better job of "getting in the mind" of the
creator of this code to understand the methodology.
- Wherever it says "To Do"
or "Source
code says..." provide better understanding/documentation
Additional
information:
- For general information about this WebSDR system -
including contact info - go to the about page (link).
- For the latest news about this system and current issues,
visit the latest news
page (link).
- For more information about this server you may contact
Clint, KA7OEI using his callsign at ka7oei dot com.
- For more information about the WebSDR project in general -
including information about other WebSDR servers worldwide and
additional technical information - go to http://www.websdr.org
Back to the Northern Utah WebSDR
landing page