Has the broker given a legitimate reason for paying you less? After watching plenty of "Judge Judy" court room cases when it comes to contracts it comes down to whats written on the piece of paper and what you've both agreed to. If nothings on paper then it becomes a gentlemen's agreement and we all know there's not to many of them left around. The good old days of spitting on your hand and shaking on an agreement are long gone.
You may also be able to go after the shipper or receiver if you name is on the carrier line on the bill of lading, and the non-recourse provision is not signed per section 7 of the Uniform Bill Lading provided there is nothing in the contract with the broker stating that the carrier will look only to the broker for payment. Billing the shipper might just get the broker to pay. The statute of limitations for collection of freight bills is 18 months.