I came across an older (2006) blog post via this HN post arguing that C/C++ are poor tools for numerical and scientific programming. It’s a bit dated, but I agree with many of Mark’s points. In particular, it is much more difficult to write efficient low-level numerical code in C than in some other languages. This is particularly true compared to, for example, Fortran or functional languages like OCaml.

However, I’m not ready to ditch C/C++ and start building my next project in one of those languages. For these decisions, I think it’s important to distinguish between low- and high-level numerical programming. This point is being hashed-out somewhat in the HN thread, but it seems that relatively few of them are building analytics software for scientific users. From my perspective, I want three (conflicting) traits in my programming language for numerical development:

1. Extremely high efficiency
2. Easy integration with data management code and libraries
3. Fast development so I can test a wide range of computational approaches

Mark argues that C is not ideal for 1, which I agree with. However, I find C excellent for 2 and not terrible for 3.

C is not ideal for the lowest-level forms of numerical development. The best low-level code (e.g. BLAS and LAPACK) is typically written in Fortran or extraordinarily tweaked, algorithmically-generated C (e.g. ATLAS). However, I’m not rewriting BLAS or LAPACK. When I need very fast code, I almost always use C/C++ with hooks to those libraries. In C++, I sometimes also use Eigen, which performs very well with relatively low verbosity. I know that the people building these libraries have done a vastly better job optimizing the lowest-level details than I could in a decade or two, so I build upon their work.

In terms of integration with other languages (I’m particularly focused on R and Python), C/C++ is excellent. There are built-in APIs and well-written interfaces to make life easier (e.g. Rcpp and Boost.Python), making development vastly easier. Fortran is alright on this point, although it usually entails more difficulties. OCaml, Lisp, etc. see far less support in the community and are generally less attractive for building reusable code.

For speed of development, C/C++ is not great. However, for much of the development process, I typically work in a higher-level language (R or Python), dropping to C/C++ once the algorithm’s overall form is fixed. Also, when working with BLAS/LAPACK/Eigen, it is quite easy to switch between different algorithms for tasks like matrix factorizations or eigendecompositions. OCaml and other modern languages can excel for speed of development, while Fortran is essentially a draw with C/C++; after all, BLAS and LAPACK are, at their core, Fortran libraries.

So, for now, I’m staying with C/C++ for high-performance numerical development. They’re not the prettiest languages, and they’re not the easiest to work with, but they are very fast and they play nicely with everything else I use. The only group of numerical developers that I’m really jealous of at this point are (ironically) those working in .Net — most of these people work in finance. They have access F# along with the full range of .Net languages and development tools, so they can drop into functional code where it’s useful and still have all of their components play together (reasonably) nicely. Ideally, we’ll see this type of integration on the open-source side in the near future.

On the positive side, the measures the authors demonstrate appear easy to use and reasonably effective for the detection of non-linear relationships between variables with moderate amounts of noise. It is also more interpretable than many other such statistics (e.g. unnormalized mutual information), has a useful relationship to R^2 in linear settings, and has very nice invariance properties under monotone univariate transformations. I am also intrigued by their definition of “equitability” for measures of association; it seems useful, but tricky to pin down when, for example, looking at different noise structures.

Now, my reservations. First (and foremost, on a technical level), their technique for estimation depends heavily on the maximal grid size B(n). This plays the role of an inverse bandwidth parameter — large values allow for finer grids. Setting B(n) too large leads to overfitting and inflated estimates of MIC, as the authors show in the supplement. To address this, the authors suggest setting B(n) to n^0.6 as a reasonable choice based on simulation studies. This is somewhat analogous to scaling results obtained for other nonparametric methods, but the authors of this paper do not attempt to find the optimal coefficient on this rate for any estimation criterion. I found this somewhat surprising given the attention devoted to optimal bandwidth selection for many existing nonparametric methods.

On a broader level, I have some cautions on the proposed methods:

• The basic approach of searching a vast amount of raw data for complex relationships can be problematic. What surfaces are often artifacts of the measurement process. Conversely, using preprocessed data can simply reveal effects of the processing rather than science. The examples used in this paper appear solid, but I worry about such issues if this approach is broadly adopted.
• There is no free lunch. This is a neat technique, but it (necessarily) has less power to detect linear relationships than, e.g., F tests in the presence of Gaussian noise. MIC is also (currently) limited to bivariate relationships; scaling it to more than two variables appears difficult. I like MIC as an addition to my toolbox, but it’s not going to replace everything else.
• Multiple testing is still tricky. The authors handle it fairly well (FDR control via Benjamini-Hochberg), but this is point of concern as the number pairwise of relationship scales as p^2. This will pose a greater challenge if this approach is pushed to relationships between more than two variables.
• Their techniques really look for a lack of independence, not any particular form of non-linearity. This is great for exploration, but it is also very limited compared to model-based approaches. MIC and related methods cannot be used directly for prediction or easily integrated with other information in a principled way like probabilistic models can.

Overall, MIC and its related methods look like a solid contribution to the literature on nonparametric EDA, but they are not a panacea. I am curious to see how they are received and developed by the rest of the statistical community.

Edit: Moved commentary below the fold