1. Rashbam’s comment appears to be straightforward. Mount Seir and the desert around it are to the south of the (biblical) land of Israel. (As NJPSC puts it: “To judge from verse 8, the Israelites were now at the southern tip of Seir-Edom, near Elath and Ezion-geber.”) The people are told to travel towards the land of Israel, i.e., to go north. Modern commentators say the same thing.
Rashbam’s comment seems to be directed against the convoluted explanation of Rashi here:

NOW TURN NORTH: Now turn along the eastern border [of Edom], from south to north, your faces (directed) towards the north. Thus they were going along the eastern border... .

In order to understand the dispute here between Rashi and Rashbam it is necessary to explain Rashi’s position.
One of the problems that Rashi and Rashbam are addressing here is the discrepancy between the story about Israel and the Edomites here in chapter 2 and the parallel story in Numbers 20:14-21. Rashi and Rashbam have very different ways of resolving that problem. (See Rashbam’s commentary ad Numbers 20:17, and note 23 there; below ad vs. 4, and note 32 there; and below ad vs. 29, and note 56 there.) Rashbam says that the Israelites did travel through some Edomite territory. Rashi says that the Israelites never entered any Edomite territory.
Rashi, accordingly, is troubled by our verse. The Israelites are, we assume, traveling to the south of Edomite territory. God tells the Israelites to go north, presumably to go through Edomite territory to the land of Israel. But according to the simplest reading of the text, the Israelites never actually do that. Rashi is troubled that, as the super-commentary Gur ’Aryeh explains, if we say that God told the Israelites to go north through Edomite territory, and they eventually were not allowed to do so, then it seems that God’s instructions actually were for naught (יהיה דבר ה' ... בחינם).
In order to avoid that interpretation, Rashi says (in his commentary to vs. 1) that the words (in vs. 1) “we skirted Mt. Seir” mean that the Israelites were marching eastward along the southern border of Edomite territory, towards the southeast corner of Edomite territory. While they were marching eastward God said to them, in our verse, that they were to “go north.” We must understand, says Rashi, that that did not mean to go north right away through Edomite territory. Rather it meant to go north once they had traversed the entire length of the southern border of Edom. Then they were to turn northward, proceeding along the eastern periphery of the Edomite territory. See Rashi’s comment to vs. 8 below, and see also Hizq. here: “The command here [to go north] did not mean [to go north] immediately.”
When God tells the Israelites in vs. 4, “You will be passing בגבול your brothers, the Edomites,” Rashi (although he refrains from commenting) presumably thinks that it means that the Israelites will be marching along the Edomite border, along the periphery of their territory.
For Rashbam, Rashi’s problem is not a problem. The Israelites did, according to Rashbam, go through some Edomite territory in their attempt to reach the land of Israel. When God says in our verse, “Go north,” it simply means to go north, through Edomite territory, in the direction of the land of Israel. When God says in vs. 4, “You will be passing בגבול your brothers, the Edomites,” it means that the Israelites are really entering into Edomite territory. See, like Rashbam, all moderns (e.g. NJPS: “You will be passing through the territory of your kinsmen, the descendants of Esau.”). See also (like Rashbam) Weinfeld (Anchor, p. 159):

“You will be passing through the territory”: Literally, “through the border,” but the Hebrew word for border, gebul, denotes territory (cf., e.g. Exod. 13:7). In the parallel story in Numbers (20:14-21), the same phrase ‘br bgbl occurs (vs. 21) and undoubtedly means “to cross the territory.”