Trappist beer is a designation of the origin of the beer, rather than a designation for the style. That is, it must come from one of the ten Trappist monasteries recognised by the International Trappist Association. There are similarities between the styles produced by these monasteries, but if one decided to make something in a completely different style it could presumably still be called a Trappist beer.
In contrast, the term abbey beer is used to refer to any beers produced in the style of the Trappist monasteries. It isn't necessarily produced by a monastery, and some beers produced by major breweries could be described as abbey beer.
It is kind of like the difference between "Champagne" and "sparkling white wine".