Abstract
The recursive solution to the Procrustes problem -with or without constraints- is thoroughly investigated. Given known matrices and , the proposed solution minimizes the square of the Frobenius norm of the difference when rows or columns are added to and . The proposed method is based on efficient strategies which reduce the computational cost by utilizing previous computations when new data are acquired. This is particularly useful in the iterative solution of an unbalanced orthogonal Procrustes problem. The results show that the computational efficiency of the proposed recursive algorithms is more significant when the dimensions of the matrices are large. This demonstrates the usefulness of the proposed algorithms in the presence of high-dimensional data sets. The practicality of the new method is demonstrated through an application in machine learning, namely feature extraction for image processing.
1. Introduction
In practice, one may be interested in finding the matrix such that where matrices and come from experiments. However, and often do not satisfy the solvability conditions and hence, the least squares solution of the difference is required [Citation15]. Specifically, the problem of approximating one given matrix with another given matrix by a transformation matrix so that the square of the difference is minimized is known as the Procrustes problem [Citation13,Citation15,Citation33]. Often, depending on the application, it is assumed that belongs to a specific class of matrices, and thus setting in this way a set of constraints to the optimization problem. The most frequent classes of matrices for is orthogonality and symmetry, and variants thereof, see, for example [Citation5,Citation11,Citation15,Citation19,Citation21,Citation29,Citation31,Citation33]. In many cases, orthogonal factorizations like the singular value decomposition (SVD), the eigenvalue decomposition (EVD) and the CS decomposition have been used to solve the Procrustes problem and variants thereof [Citation8,Citation15,Citation19,Citation22,Citation31,Citation33, pp. 327–328].
The application of the Procrustes problem in factor analysis has a long history [Citation10,Citation13,Citation15,Citation28]. It also appears in numerical analysis problems for the solution of partial differential equations, in multidimensional scaling, in growth curve modelling, in scientific computing, in computer vision, in image processing, in system and control theory, in the analysis of space structures and in aerospace engineering for spacecraft attitude determination [Citation12,Citation23,Citation24,Citation26,Citation27,Citation32,Citation38,Citation41,Citation43]. Moreover, in the case where is the identical matrix the problem becomes a matrix nearness problem with many applications in statistical and financial modelling and in theoretical computer science, see, for example [Citation20,Citation36,Citation39].
The recursive solution to a least squares problem is needed when the experiment is conducted repeatedly and as a result the given matrices are updated with new arriving data regularly. Also, in high dimensional settings, the matrices and are very large and it may not be possible to treat all data at once or the computational cost of processing them may be significantly expensive. In this case, a sequential procedure which splits and into sub-matrices of smaller dimensions and then proceeds by gradually incorporating the sub-matrices into the least squares solution of is essential. Recursive least squares is often needed in many problems of different areas like engineering, statistics, econometrics and finance [Citation7,Citation9,Citation17,Citation18,Citation42]. A recursive algorithm reduces the computational cost and also the storage requirements for large matrices.
Herein, the recursive least squares solution to the matrix equation when and are known matrices is investigated in depth. Namely, the recursive solution to the Procrustes problem is examined. The use of the QR decomposition is examined when there are no constraints on . Also, the problems of minimizing the difference when is orthogonal and when is symmetric are also considered and their recursive least squares solution using the eigenvalue and singular value decompositions is explored in depth. When constraints are imposed on , the method of Lagrange multipliers is used to solve the optimization problem. The proposed solution, in each case, is the matrix which minimizes the square of the Frobenious norm of . It is an exact solution in the sense that is explicitly determined and does not comprise any arbitrary elements. The recursive numerical solution proposed does not require the matrices be full rank.
Throughout this paper, denotes the Frobenius norm. Also, for known matrices and , when computing partial derivatives [Citation25, p. 201] the following properties are used: (1) (1) The paper is organized as follows. Section 2 introduces the general Procrustes problem where no assumption is made for the solution matrix . The problem is solved using the QR decomposition and then the recursive solution is presented. Section 3 considers the orthogonal Procrustes problem where the solution matrix is orthogonal and Section 4 derives the solution to the symmetric Procrustes problem when is assumed to be symmetric. Section 5 presents computational results and finally, in Section 6 we conclude and discuss future work.
2. Numerical solution to the general procrustes problem
Consider the problem of finding a matrix so that the known matrix is approximated by matrix where is also known. That is, a solution to the matrix equation (2) (2) is required. The least squares approximation problem to be solved is given by (3) (3) where On using (Equation1(1) (1) ) partial differentiation yields (4) (4) whence (5) (5) When is non-singular the solution to (Equation5(5) (5) ) is given by [Citation14]. However, may be ill-conditioned and in this case inverting the matrix may give inaccurate results. In the case where is singular the latter will fail to give a solution for . A numerically stable method to obtain is to use the QR decomposition (QRD) of , namely (6) (6) Then and hence and . Therefore, . When is rank deficient, the procedure to obtain is similar to the above, namely (Equation6(6) (6) ). In this case, a complete QRD can be computed to triangularize [Citation1].
2.1. Recursive solution
We next consider the recursive solution of the Procrustes problem when the matrices and are updated. Without loss of generality, suppose that and are augmented with the addition of a single row; namely, (7) (7) where are as in (Equation2(2) (2) ) and represent new data points. Then, the updated Procrustes problem, based on and , requires the solution of the least squares problem (8) (8) when (Equation3(3) (3) ) has already been solved (see (Equation4(4) (4) )–(Equation6(6) (6) )). The efficient solution of (Equation8(8) (8) ) requires that previous computations from the solution of (Equation3(3) (3) ) be utilized. Namely, where , and are as in (Equation6(6) (6) ) and is the n-dimensional row vector of ones. Consider now the updating QRD (9) (9) We then have that and therefore it follows that .
In many cases, it is possible that the matrices and are updated with a single column, namely where denote new variables which become available after (Equation3(3) (3) ) has been solved. As a result, the solution to the updated Procrustes problem (10) (10) needs to be computed. By utilizing efficiently previous computations from the solution of (Equation3(3) (3) ), (Equation10(10) (10) ) is written as where is from the QRD of in (Equation6(6) (6) ). The column-updating QRD that needs to be computed is then given by where is an orthogonal transformation that eliminates all but the first element of and is a scalar. Hence, the updated Procrustes problem (Equation10(10) (10) ) becomes (11) (11) where The solution to problem (Equation11(11) (11) ) is given by .
3. The orthogonal procrustes problem
The orthogonal Procrustes problem (OPP) is that of minimizing the sum of the squared error of the difference matrix when the unknown matrix is orthogonal. The constraint is imposed by using the method of Lagrange multipliers; the matrices and need not be full rank [Citation33]. The constrained optimization problem is then given by (12) (12) where and is orthogonal. Herein, we are most interested in cases where m<n. To find the solution to (Equation12(12) (12) ), we consider the Lagrangian function which is equivalently written as (13) (13) where is the symmetric matrix of Lagrange multipliers [Citation15]. Partial differentiation of (Equation13(13) (13) ) yields (14) (14) On setting (Equation14(14) (14) ) equal to zero and using matrix notation, (Equation14(14) (14) ) can be written as or equivalently, as (15) (15) where is the qth column of , , are the jth columns of and , respectively, and is the qth row of . From (Equation15(15) (15) ), , therefore and, as observed earlier, is symmetric. Now on post-multiplying (Equation15(15) (15) ) by gives which implies that , whence . Therefore, (16) (16) Furthermore, let and consider the following two matrices and Matrices are symmetric and thus they are diagonalizable and their eigenvalue decomposition (EVD) exists, that is, (17a) (17a) and (17b) (17b) where are orthogonal. Additionally, since they are of the form and they have the same eigenvalues. Now on using (Equation16(16) (16) ) it follows that and from (Equation17a(17a) (17a) ) and (Equation17b(17b) (17b) ) we now have that which implies that , where is as in (Equation17a(17a) (17a) ). Therefore, the solution to the least squares problem (Equation12(12) (12) ) is given by Finally, a sufficient condition which guaranties the uniqueness of the solution and that the argument in (Equation12(12) (12) ) is minimized requires that all the diagonal elements of the matrix in (Equation17a(17a) (17a) ) and (Equation17b(17b) (17b) ) are non-negative [Citation15,Citation33]. In essence, this condition determines the orientation of the orthogonal matrices and and is specified by the Eckart–Young decomposition (see [Citation10]) of , namely Notice that in the case of symmetric orthogonality the procedure is the same but at the last step the symmetry of needs to be taken into account. Namely, the sum of is not explicitly computed since only the upper or lower part of needs to be determined. This results in a dimensional reduction of the solution matrix from to [Citation40].
3.1. Recursive solution to the orthogonal procrustes problem
Suppose that an updated orthogonal Procrustes problem needs to be solved when new data become available. Without loss of generality it is assumed that the original OPP (Equation12(12) (12) ) needs to be re-solved when appending a new single row of data in matrices and . That is, let the row updated matrices and be as in (Equation7(7) (7) ), where are as in (Equation12(12) (12) ) with . The updated orthogonal Procrustes problem based on and requires that the solution of the following least squares problem be derived: (18) (18) where . The solution to the least squares problem (Equation18(18) (18) ) is obtained by using the method of Lagrange multipliers, namely (19) (19) which yields the first order condition on using (Equation1(1) (1) ). In a way similar to (Equation12(12) (12) ), (Equation19(19) (19) ) has the solution (20) (20) where and are the orthogonal matrices of the EVD of and , respectively.
The recursive solution of the OPP presumes that previous computations in (Equation17a(17a) (17a) ) from the solution of the original Procrustes problem (Equation12(12) (12) ) are efficiently utilized. Consider the recursive computation of the EVD of , namely (21) (21) on using (Equation17a(17a) (17a) ). Therefore, the recursive solution of an orthogonal Procrustes problem becomes a modified symmetric matrix eigenvalue problem [Citation3,Citation4,Citation16]. In particular, (Equation21(21) (21) ) implies three rank-1 modifications of the EVD of the matrix in (Equation17a(17a) (17a) ). That is, the EVD of is obtained recursively in three main steps. First, (Equation21(21) (21) ) is written as (22) (22) where and is a scalar since and are row vectors. Second, the following EVD is derived (23) (23) and, since is a symmetric matrix, it is diagonalizable. Third, on using (Equation22(22) (22) ), (Equation23(23) (23) ) becomes (24) (24) where with , being the eigenvalues of . Consider now the sequential updating of the diagonal matrix in two steps. The first step computes the EVD of , where is orthogonal and is diagonal. The second step computes the EVD of , where is orthogonal and is the diagonal matrix with elements the eigenvalues of . That is, , where and is as in (Equation17b(17b) (17b) ). Repeating the procedure at steps (Equation21(21) (21) )–(Equation24(24) (24) ) for will derive its EVD recursively and will therefore give , where , is defined in (Equation17a(17a) (17a) ) and are computed in a way similar to that for (Equation24(24) (24) ). Therefore, the updated solution (Equation20(20) (20) ) to the Procrustes problem (Equation18(18) (18) ) has been derived.
When extra columns are added to and , that is when (25) (25) an updated orthogonal Procrustes problem of larger dimensions needs to be solved. Namely, (26) (26) where has been augmented by k columns and k rows. In a similar way as for (Equation18(18) (18) ), the solution of (Equation26(26) (26) ) is given by . It is obtained recursively, by updating the original orthogonal decompositions as in (Equation24(24) (24) ). Notice that the addition of extra columns implies a rank-k updating of the EVD decomposition.
4. The symmetric procrustes problem
Consider the problem of minimizing the sum of squared error of the difference matrix when the unknown matrix is symmetric. The constrained optimization problem is given by where is a symmetric matrix. As in the case of the OPP (Equation12(12) (12) ), the matrices and are not necessarily full rank. Using the method of Lagrange multipliers, the problem becomes that of finding the matrix which minimizes On using (Equation1(1) (1) ) partial differentiation yields which is set to zero. Now let the matrix , which is skew-symmetric, that is . It follows that which yields the Lyapunov equation (27) (27) Since is symmetric, it is therefore a diagonalizable matrix. That is, there is an orthogonal matrix and a diagonal matrix such that (28) (28) where has columns the eigenvectors of and has diagonal elements the eigenvalues of . Using (Equation27(27) (27) ), (Equation28(28) (28) ) becomes (29) (29) where and . By utilizing the diagonal structure of , it follows that where . The solution is then given by A necessary and sufficient condition for the uniqueness of , since is positive definite, is that all the eigenvalues of have a negative real part, that is, is a stable matrix [Citation2,Citation34,Citation35].
4.1. Recursive solution to the symmetric procrustes problem
Consider now the case where the matrices and are augmented by the addition of an extra row as in (Equation7(7) (7) ). The updated symmetric Procrustes problem requires the solution of the optimization problem where and are defined as in (Equation7(7) (7) ). Using the Lagrange multipliers and the analysis for the symmetric Procrustes problem as in the previous section, the solution is obtained from the Lyapunov equation (30) (30) by computing the EVD of . To recursively solve (Equation30(30) (30) ) observe that on using (Equation28(28) (28) ) yields Therefore, the sequential updating of requires one rank-1 update, namely, , whence with . The Lyapunov equation (Equation30(30) (30) ) then becomes (31) (31) where and . The solution to (Equation31(31) (31) ) is obtained in a way similar to that for (Equation29(29) (29) ).
5. Computational remarks
To amplify the practical usability of the proposed method, the theoretical complexity analysis of the new algorithms has been studied. Consider the general Procrustes problem when no constraint is imposed to the least squares solution matrix . Let the matrices as in (Equation7(7) (7) ) and assume that the QRD of in (Equation6(6) (6) ) is available. The complexity of computing the QRD of afresh is floating point operations (flops) and that of applying these orthogonal transformations to matrix is flops. The complexities of computing the updating QRD in (Equation9(9) (9) ) in order to update and require and flops, respectively. As a result, to compute the QRD of and to apply the orthogonal transformations to will always be computationally more demanding than computing it recursively utilizing previous calculations. The computational efficiency of the recursive method compared to computing the QRD from scratch is, approximately, given by .
The solution of the orthogonal Procrustes problem is derived from the EVD of the square matrices and . However, in practice the SVD of is computed since and . When and are modified, the computationally efficient solution to the orthogonal Procrustes problem requires that the SVD of is computed recursively. Therefore, in the recursive solution of the orthogonal and symmetric Procrustes problems, updating SVD decompositions are computed which provide equivalent results as if the EVD were computed. The SVD of an matrix requires flops in the first stage and flops in the second stage where k is the number of iterations required to compute each singular value. Herein, it is assumed that and it holds that .
The algorithms employ a low-rank modification strategy for the recursive updating of the SVD as in [Citation3]. Algorithms 5.1–5.3 summarize the main computational steps for the solution of the orthogonal Procrustes problem and the recursive proposed solution when new rows or columns of data are added, respectively.
To evaluate further the new algorithms, experiments based on synthetic and real data have been conducted. The computational times of the new algorithms have been compared with the algorithm that solves the same problem afresh in order to obtain the efficiency ratio.
Table presents the execution times (in CPU seconds) of Algorithm 5.1 compared with Algorithm 5.3 when m = 50, 100, 250, 500 rows and n = 1000, 5000 and a single column of data is added to and , that is k = 1. The results show that keeping n fixed and increasing m reduces the computational efficiency. However, comparing Panel A and B shows that the efficiency is more significant when n increases. Table presents the execution times (in CPU seconds) of the Algorithms 5.1 and 5.2 when m = 100, n = 1000 and when m = 500, n = 10, 000 with a variable number k of new columns are added to and . All the times presented are the average times after solving the same OPP (afresh or recursively) 100 times. The computational results in Table show that as the number of dimensions increase the computational efficiency also increases. Furthermore, the results suggest that the proposed algorithm for the addition on new column is more efficient when a small number of extra columns is included. Comparing the results in Panel A and B of Table we see that the efficiency of the proposed recursive method increases when the dimensions of the matrices in the original OPP also increase.
Many a time, the solution of an OPP is applied in the development of algorithms for face recognition, see for example [Citation6,Citation30,Citation37,Citation43]. The algorithm proposed in [Citation43] for feature extraction requires to solve -until convergence- a series of updating OPPs as in (Equation12(12) (12) ) where is the data matrix and is the class indicator matrix after they have both been centred. The processing of an image starts by considering their pixels as the elements of a matrix and then converting each matrix (or each image) to a column vector. Each row in has length equal to the feature number of the images. When a new image becomes available and is added to the database, a row will be added to . The face recognition algorithm will then have to process the extra image, that is the additional row of the data matrix . This is equivalent to row updating as in (Equation18(18) (18) ). Herein, Algorithm 5.3 is employed to show the computational efficiency of utilizing previous computations when a series of updating OPPs is solved after augmenting the data matrix.
In Table the algorithm that estimates the problems afresh and the recursive algorithm are compared. It is assumed that there are m = 2000, 5000, 10, 000, 15, 000, 20, 000 images available which have been cropped and re-sized to and pixels, that is n = 256, 1024. The run times (in CPU seconds) when an extra image or alternatively an extra row of data is included 100 times are presented. The results show that when new rows of data are added to the problem the efficiency increases as the number of rows increase. The reason for this is the fact that the recursive algorithm does not depend on m, the number of rows of matrices and . Instead, it depends on the number of columns n of and and on the number of rows l added to the model. This is also why the timings of Algorithm 5.3 are the same when n and l are fixed.
Overall, the results show that the computational efficiency of the proposed recursive algorithm increases when the dimensions of the matrices (data) increase. The computational efficiency is significantly more important when a small number of rows or columns is amended -compared to the original matrices- that is, the modification is low-rank. This demonstrates the practicability of the proposed method when solving sequentially OPPs with small modifications in the underlying dataset. The proposed methods are particularly usable when the matrices involved are large-scale and the data are high-dimensional. All the reported computational results were performed on a 64-bit 1.80 GHz Core(TM) i7-8550U processor and 16.0 GB RAM using R (version 3.6.1).
6. Conclusions and future work
The recursive least squares solutions to the matrix equation AX = B when new rows or columns of data are added to data matrices A and B is thoroughly investigated. A computationally efficient algorithm which is based on the singular value decomposition is proposed. Computational results are presented for synthetic data and also for a machine learning application based on feature extraction for face recognition. The recursive solution to the symmetric Procrustes problem when including extra data is also investigated. The experimental results suggest that the proposed algorithms are computationally more efficient when the matrices are high-dimensional and when they are augmented with a small number of rows or columns.
The extension of the proposed method to include other special classes of matrices like reflexive and anti-reflexive, Stiefel matrices or Toeplitz matrices merits further investigation. Future work will also consider the solution to the Procrustes problem with regularization constraints after modifying the matrices with the addition or deletion of data.
Disclosure statement
No potential conflict of interest was reported by the author(s).
References
- E. Anderson, Z. Bai, and J. Dongarra, Generalized QR factorization and its applications, Linear Algebra Appl. 162–164(0) (1992), pp. 243–271.
- R.H. Bartels and G.W. Stewart, Solution of the matrix equation AX + XB = C [F4], Commun. ACM 15(9) (1972), pp. 820–826.
- M. Brand, Fast low-rank modifications of the thin singular value decomposition, Linear Algebra Appl. 415(1) (2006), pp. 20–30.
- J.R. Bunch and C.P. Nielsen, Updating the singular value decomposition, Numer. Math. 31(2) (1978), pp. 111–129.
- J.R. Cardoso and K. Ziẹtak, On a sub-Stiefel procrustes problem arising in computer vision, Numer. Linear Algebra Appl. 22(3) (2015), pp. 523–547.
- D. Chen, X. Cao, F. Wen, and J. Sun. Blessing of dimensionality: High-dimensional feature and its efficient compression for face verification, in 2013 Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Portland, OR, USA, 2013, pp. 3025–3032.
- R. Christensen, L.M. Pearson, and W. Johnson, Case-deletion diagnostics for mixed models, Technometrics 34(1) (1992), pp. 38–45.
- K. Chu, Singular value and generalized singular value decompositions and the solution of linear matrix equations, Linear Algebra Appl. 88 (1987), pp. 83–98.
- R.D. Cook, Influential observations in linear regression, J. Am. Stat. Assoc. 74(365) (1979), pp. 169–174.
- C. Eckart and G. Young, The approximation of one matrix by another of lower rank, Psychometrika 1(3) (1936), pp. 211–218.
- L. Eldén and H. Park, A procrustes problem on the Stiefel manifold, Numer. Math. 82(4) (1999), pp. 599–619.
- Y. Gong, S. Lazebnik, A. Gordo, and F. Perronnin, Iterative quantization: A procrustean approach to learning binary codes for large-scale image retrieval, IEEE Trans. Pattern Anal. Mac. Intell. 35(12) (2013 Dec), pp. 2916–2929.
- J.C. Gower, Generalized procrustes analysis, Psychometrika 40(1) (1975), pp. 33–51.
- J.C. Gower, Procrustes methods, Interdiscip. Rev. Comput. Stat. 2(4) (2010), pp. 503–508.
- B.F. Green, The orthogonal approximation of an oblique structure in factor analysis, Psychometrika 17(4) (1952), pp. 429–440.
- M. Gu and S.C. Eisenstat, Downdating the singular value decomposition, SIAM J. Matrix Anal. Appl. 16(3) (1995), pp. 793–810.
- S. Hadjiantoni and E.J. Kontoghiorghes, Estimating large-scale general linear and seemingly unrelated regressions models after deleting observations, Stat. Comput. 27(2) (2017), pp. 349–361.
- S. Hadjiantoni and E.J. Kontoghiorghes, A recursive three-stage least squares method for large-scale systems of simultaneous equations, Linear Algebra Appl. 536 (2018), pp. 210–227.
- N.J. Higham, The symmetric procrustes problem, BIT Numer. Math. 28(1) (1988), pp. 133–143.
- N.J. Higham and N. Strabić, Bounds for the distance to the nearest correlation matrix, SIAM J. Matrix Anal. Appl. 37(3) (2016), pp. 1088–1102.
- D. Hua, On the symmetric solutions of linear matrix equations, Linear Algebra Appl. 131 (1990), pp. 1–7.
- A.-P. Liao and Y. Lei, Least-squares solution with the minimum-norm for the matrix equation (AXB, GXH) = (C, D), Comput. Math. Appl. 50(3) (2005), pp. 539–549.
- X. Liu, Hermitian and non-negative definite reflexive and anti-reflexive solutions to AX=B, Int. J. Comput. Math. 95(8) (2018), pp. 1666–1671.
- Y.H. Liu, Ranks of least squares solutions of the matrix equation AXB=C, Comput. Math. Appl. 55(6) (2008), pp. 1270–1278.
- J.R. Magnus and H. Neudecker, Matrix Differential Calculus with Applications in Statistics and Econometrics, 3rd ed., Wiley, Chichester, UK, 2007.
- F.L. Markley, Attitude determination using vector observations and the singular value decomposition, J. Astronaut. Sci. 36(3) (1988), pp. 245–258.
- C. Meng, X. Hu, and L. Zhang, The skew-symmetric orthogonal solutions of the matrix equation AX=B, Linear Algebra Appl. 402 (2005), pp. 303–318.
- C.I. Mosier, Determining a simple structure when loadings for certain tests are known, Psychometrika 4(2) (1939), pp. 149–162.
- X.Y. Peng, X.Y. Hu, and L. Zhang, The reflexive and anti-reflexive solutions of the matrix equation AHXB=C,J. Comput. Appl. Math. 200(2) (2007), pp. 749–760.
- Y. Peng, W. Kong, and B. Yang, Orthogonal extreme learning machine for image classification, Neurocomputing 266 (2017), pp. 458–464.
- Y. Qiu and A. Wang, Solving balanced procrustes problem with some constraints by eigenvalue decomposition,J. Comput. Appl. Math. 233(11) (2010), pp. 2916–2924.
- M.B. Rubin and D. Solav, Unphysical properties of the rotation tensor estimated by least squares optimization with specific application to biomechanics, Int. J. Eng. Sci. 103 (2016), pp. 11–18.
- P.H. Schönemann, A generalized solution of the orthogonal procrustes problem, Psychometrika 31(1) (1966),pp. 1–10.
- V. Simoncini, Computational methods for linear matrix equations, SIAM Rev. 58(3) (2016), pp. 377–441.
- J. Snyders and M. Zakai, On nonnegative solutions of the equation AD+DA′=−C, SIAM J. Appl. Math. 18(3) (1970), pp. 704–714.
- Y. Sun and L. Vandenberghe, Decomposition methods for sparse matrix nearness problems, SIAM J. Matrix Anal. Appl. 36(4) (2015), pp. 1691–1717.
- Y. Tai, J. Yang, Y. Zhang, L. Luo, J. Qian, and Y. Chen, Face recognition with pose variations and misalignment via orthogonal procrustes regression, IEEE Trans. Image Process. 25(6) (2016), pp. 2673–2683.
- Y. Tian and Y. Takane, On consistency, natural restrictions and estimability under classical and extended growth curve models, J. Stat. Plan. Inference 139(7) (2009), pp. 2445–2458.
- J.A. Tropp, A. Yurtsever, M. Udell, and V. Cevher, Practical sketching algorithms for low-rank matrix approximation, SIAM J. Matrix Anal. Appl. 38(4) (2017), pp. 1454–1485.
- W.J. Vetter, Vector structures and solutions of linear matrix equations, Linear Algebra Appl. 10(2) (1975),pp. 181–188.
- G. Wahba, A least squares estimate of satellite attitude, SIAM Rev. 7(3) (1965), pp. 409–409.
- P.C. Young, Recursive Estimation and Time-Series Analysis, 2nd ed., Springer-Verlag, Berlin Heidelberg, 2011.
- H. Zhao, Z. Wang, and F. Nie, Orthogonal least squares regression for feature extraction, Neurocomputing 216 (2016), pp. 200–207.