1. The cantillation sign for the word דרך is standardly called a yetiv. At the end of this comment, Rashbam calls it a shofar mahpakh (or perhaps shofar mehuppakh). The term is unusual but the meaning is clear. What we today call a yetiv was known by other names at different times in history (shofar yetiv or yetiv mahpakh or mashpel or shofar muqdam. See S. Weinfeld’s Ṭa‘ame ha-miqra’ [Jerusalem, 1972]). The name shofar mahpakh does appear once in Rashi’s Talmud commentary (to Ber. 62a, s.v. טעמי תורה), but it is difficult to tell there to which cantillation sign Rashi is referring. Here Rashi calls this cantillation sign a mashpel.

2. Rashbam is basically correct that a pashṭa turns into a yetiv in a word with stress on the first syllable. However, his rule should be refined slightly: A pashṭa turns into a yetiv in a word with stress on the first syllable, as long as there is no conjunctive cantillation sign (ṭa‘am mesharet) on the previous word.
It is surprising to find Rashbam discussing the rules of cantillation signs, not standardly a topic of discussion in his Torah commentary. The order of cantillation signs here – pashṭa-yetiv-zaqef qaṭon – is unusual, but when it appeared previously in the Torah (e.g. in Leviticus 25:8), Rashbam did not comment on it.
Most likely, Rashbam is taking issue with Rashi’s explanation of the verse. To understand what Rashbam is saying it is necessary to understand Rashi’s interpretation. Rashi argues that the disjunctive pashṭa sign on the word אחרי precludes reading that word in the usual way as a preposition meaning “after” or “beyond.” If that were the meaning, the word אחרי, Rashi reasons, would have to have a conjunctive sign. So, he argues, the verse should be understood as follows:

“They are on the other side of the Jordan,” but not close to the Jordan. They are rather “אחרי – far away.” And in which direction are they from the Jordan? They are “דרך מבוא השמש – towards the setting sun,” that is to the west of the Jordan.

(Mendelssohn follows Rashi’s understanding: יענזייטש דש ירדנש וייט היניבר, געגן זאנן אונטרגאנג.) Rashi writes specifically that the cantillation “מוכיח שהוא שני דברים שננקדו בשני טעמים – proves that there are two separate ideas here because they [the words אחרי and דרך] were marked with two different disjunctive cantillation signs.”
Rashbam at the simplest level takes issue with Rashi’s statement that there are “two different cantillation signs” here. There are not, he says. Both the word אחרי and the word דרך are marked with the same sign, a pashṭa. However, a pashṭa automatically changes to a yetiv in certain circumstances (as explained above in this footnote). Rashbam’s comment, then, might simply be a gloss on what he considers to be Rashi’s infelicitous word choice.
However, it is possible that Rashbam is also disputing Rashi’s interpretation of the verse. Luzzatto was the first (as far as I can to determine) to point out that Rashi’s incomplete understanding of all the rules of cantillation signs (“since the rules of the cantillation signs were not well known in his days”) leads him to erroneous conclusions here. Luzzatto explains that אחרי is a preposition meaning “after” or “beyond.” It could not have a conjunctive cantillation sign, since conjunctive cantillation signs are used only to connect a word to the following one word. Since אחרי is connected to a following three-word phrase – דרך מבוא השמש – the rules call for a disjunctive sign. Since דרך is not connected to the following one word but to the following two-word phrase – מבוא השמש – it too needs a disjunctive cantillation sign. Since אחרי “governs” a larger phrase than דרך does, אחרי requires a stronger disjunctive sign. According to the rules of the cantillation signs, when one disjunctive sign is used on two words in a row, its first instance of use represents the “stronger” disjunction. So having two pashṭa signs in a row here (or their equivalent, a pashṭa followed by a yetiv) is an unremarkable way of “punctuating” our verse and does not support Rashi’s unusual interpretation. See also Kogut’s reformulation of Luzzatto’s argument in Ha-miqra’ ben ṭe‘amim lefarshanut, pp. 50-52. They both cite the identical pattern of cantillation signs in the phrase ימי שבע שבתות השנים (Leviticus 25:8) to prove their point.
Rashbam’s next comment makes it clear (see note 7) that he disputes Rashi’s understanding of the phrase אחרי דרך מבוא בשמש. It is possible that when he points out that our verse has a cantillation pattern that is equivalent to pashṭa - pashṭa, he is saying, like Luzzatto, that Rashi’s interpretation is not supported by the proof that Rashi adduced.

3. So also Rashi. Cf. however Sifre 56 and Sot. 33b, where the phrase is interpreted as meaning the place where the sun rises, i.e., the east. See footnote 9 on p. 123 in Finkelstein’s edition of Sifre.

4. As pointed out in note 5, Rashi reads our phrase as meaning “on the other side of the Jordan, far beyond, in the direction of the setting sun.” Rashbam explains that it means more simply “on the other side of the Jordan, beyond the west road.” Modern translations either explain like Rashbam (e.g. NJPS) or like Ibn Ezra (e.g. Everett Fox: “along [= אחרי] the path of the setting sun”), but not like Rashi.