This is LaTeX code:

[latex]
\left\langle {{\mathbf{A}}}
 \mathrel{\left | {\vphantom {{\mathbf{A}} {\mathbf{B}}}}
 \right. \kern-\nulldelimiterspace}
 {{\mathbf{B}}} \right\rangle : = \operatorname{tr} \left( {{\mathbf{B}}^* {\mathbf{A}}} \right) = \operatorname{tr} \left( {{\mathbf{AB}}^* } \right)
[/latex]