Frank,
this is a nice example of combining 2 FSMs to create a
bigger one that describes the overall system:
The DQPSK can be described as an FSM with inputs x and outputs y.
Both x,y are integers (not complex numbers).
Then the ISI channel can be thought of as another FSM with input
y and outputs z.
These 2 FSMs can be combined into a bigger one (more states)
with input x and output z (y has dissapeared from the description).
Then the output z is mapped to the actual waveform (or in the case of
DQPSK/ISI) the actual noiseless vector that is the combination of the
transmitted signal and ISI channel.
Regarding incorporating the known signals there are two issues:
The first 5 samples corresponding to the first 5 known signals can be
discarded since they do not carry any info for decoding (they will only
determine the starting state in the VA).
The last 5 signals are important and need to be processed as they carry
information about the info signals through the ISI tails. For this you
will either need to write your own “trellis_metrics_" function and use
the standard "trellis_viterbi_” block, or
change a bit the “trellis_viterbi_combined_**” block and in
particular the implementation of the “calc_metric” function.
I understand this is a high level description of what needs to be done,
but it seems you know the details.
If you need more help with the details let me know.
Achilleas