As you can see mat has ncol = 4 and vecteur has a length of 3. I thought from mathematics the # of columns of the matrix needed to match the # of rows of a vector in order for the multiplication to take place. There's another instance of the same phenomenon on Multiply Matrix by Vector in R - GeeksforGeeks

Would someone explain to me how I'm either misunderstanding the mathematical principal or how R handles the non-mathematical multiplication of a matrix by a vector?

In your example, vecteur is a 1 x 3 matrix (row vector) and mat is 3 x 4, so the product is well defined both mathematically and in R. What happens if you reverse the order of multiplication?

It seems my brain inverted the multiplication because I had only seen matrices multiplied by vectors. It seems it uses the matrices multiplied by matrices formula with the 1st matrix of a 1-column variety.
My verification:

Yes, the definition of multiplication by a matrix (in both mathematics and R), uses the matrix-matrix multiplication algorithm. This requires that a vector be interpreted as a one-row or one-column matrix, depending on the order of multiplication, and that number of columns of the left matrix equal the number of rows of the right matrix.