Oxford-Tencent Collaboration on Large Scale Machine Learning

This project involving Yee Whye Teh and Dino Sejdinovic aims to explore novel methods for machine learning, with particular foci on deep generative models and hyperparameter optimization. There will be several topics of investigation in these fields that will cut across various aspects of large-scale machine learning: efficient computation for working large datasets; scalable and expressive models that can extract as much information as possible from the available data; large-scale and heterogeneous computational substrates (e.g. GPUs, multicore systems, networked clusters) and theoretical foundations. The project is funded by Tencent AI Lab.

Publications

2021

• J. Xu , H. Kim , T. Rainforth , Y. W. Teh , Group Equivariant Subsampling, in Neural Information Processing Systems (NeurIPS), 2021.
  
  Project: tencent-lsml

  Subsampling is used in convolutional neural networks (CNNs) in the form of pooling or strided convolutions, to reduce the spatial dimensions of feature maps and to allow the receptive fields to grow exponentially with depth. However, it is known that such subsampling operations are not translation equivariant, unlike convolutions that are translation equivariant. Here, we first introduce translation equivariant subsampling/upsampling layers that can be used to construct exact translation equivariant CNNs. We then generalise these layers beyond translations to general groups, thus proposing group equivariant subsampling/upsampling. We use these layers to construct group equivariant autoencoders (GAEs) that allow us to learn low-dimensional equivariant representations. We empirically verify on images that the representations are indeed equivariant to input translations and rotations, and thus generalise well to unseen positions and orientations. We further use GAEs in models that learn object-centric representations on multi-object datasets, and show improved data efficiency and decomposition compared to non-equivariant baselines.

  @inproceedings{xu2021group,
    title = {Group Equivariant Subsampling},
    author = {Xu, Jin and Kim, Hyunjik and Rainforth, Tom and Teh, Yee Whye},
    booktitle = {Neural Information Processing Systems (NeurIPS)},
    year = {2021}
  }
  

2020

• A. Foster , M. Jankowiak , M. O’Meara , Y. W. Teh , T. Rainforth , A Unified Stochastic Gradient Approach to Designing Bayesian-Optimal Experiments, International Conference on Artificial Intelligence and Statistics (AISTATS), 2020.
  
  Project: tencent-lsml

  We introduce a fully stochastic gradient based approach to Bayesian optimal experimental design (BOED). This is achieved through the use of variational lower bounds on the expected information gain (EIG) of an experiment that can be simultaneously optimized with respect to both the variational and design parameters. This allows the design process to be carried out through a single unified stochastic gradient ascent procedure, in contrast to existing approaches that typically construct an EIG estimator on a pointwise basis, before passing this estimator to a separate optimizer. We show that this, in turn, leads to more efficient BOED schemes and provide a number of a different variational objectives suited to different settings. Furthermore, we show that our gradient-based approaches are able to provide effective design optimization in substantially higher dimensional settings than existing approaches.

  @article{foster2020unified,
    title = {{A Unified Stochastic Gradient Approach to Designing Bayesian-Optimal Experiments}},
    author = {Foster, Adam and Jankowiak, Martin and O'Meara, Matthew and Teh, Yee Whye and Rainforth, Tom},
    journal = {International Conference on Artificial Intelligence and Statistics (AISTATS)},
    year = {2020}
  }
  

• J. Xu , J. Ton , H. Kim , A. R. Kosiorek , Y. W. Teh , MetaFun: Meta-Learning with Iterative Functional Updates, in International Conference on Machine Learning (ICML), 2020.
  
  Project: tencent-lsml

  We develop a functional encoder-decoder approach to supervised meta-learning, where labeled data is encoded into an infinite-dimensional functional representation rather than a finite-dimensional one. Furthermore, rather than directly producing the representation, we learn a neural update rule resembling functional gradient descent which iteratively improves the representation. The final representation is used to condition the decoder to make predictions on unlabeled data. Our approach is the first to demonstrates the success of encoder-decoder style meta-learning methods like conditional neural processes on large-scale few-shot classification benchmarks such as miniImageNet and tieredImageNet, where it achieves state-of-the-art performance.

  @inproceedings{xu2019metafun,
    title = {MetaFun: Meta-Learning with Iterative Functional Updates},
    author = {Xu, Jin and Ton, Jean-Francois and Kim, Hyunjik and Kosiorek, Adam R and Teh, Yee Whye},
    booktitle = {International Conference on Machine Learning (ICML)},
    year = {2020}
  }
  

2019

• H. Law , P. Zhao , L. Chan , J. Huang , D. Sejdinovic , Hyperparameter Learning via Distributional Transfer, Advances in Neural Information Processing Systems (NeurIPS), to appear, 2019.
  
  Project: tencent-lsml

  Bayesian optimisation is a popular technique for hyperparameter learning but typically requires initial ’exploration’ even in cases where potentially similar prior tasks have been solved. We propose to transfer information across tasks using kernel embeddings of distributions of training datasets used in those tasks. The resulting method has a faster convergence compared to existing baselines, in some cases requiring only a few evaluations of the target objective.

  @unpublished{LawZhaHuaSej2018,
    author = {Law, H.C.L. and Zhao, P. and Chan, L. and Huang, J. and Sejdinovic, D.},
    title = {{{Hyperparameter Learning via Distributional Transfer}}},
    booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},
    pages = {to appear},
    year = {2019}
  }
  

• J. Ton , L. Chan , Y. W. Teh , D. Sejdinovic , Noise Contrastive Meta-Learning for Conditional Density Estimation using Kernel Mean Embeddings, ArXiv e-prints:1906.02236, 2019.
  
  Project: bigbayes tencent-lsml

  Current meta-learning approaches focus on learning functional representations of relationships between variables, i.e. on estimating conditional expectations in regression. In many applications, however, we are faced with conditional distributions which cannot be meaningfully summarized using expectation only (due to e.g. multimodality). Hence, we consider the problem of conditional density estimation in the meta-learning setting. We introduce a novel technique for meta-learning which combines neural representation and noise-contrastive estimation with the established literature of conditional mean embeddings into reproducing kernel Hilbert spaces. The method is validated on synthetic and real-world problems, demonstrating the utility of sharing learned representations across multiple conditional density estimation tasks.

  @unpublished{TonChaTehSej2019,
    author = {Ton, Jean-Francois and Chan, Lucian and Teh, Yee Whye and Sejdinovic, Dino},
    title = {{{Noise Contrastive Meta-Learning for Conditional Density Estimation using Kernel Mean Embeddings}}},
    journal = {ArXiv e-prints:1906.02236},
    year = {2019}
  }
  

2018

• J. Chen , J. Zhu , Y. W. Teh , T. Zhang , Stochastic Expectation Maximization with Variance Reduction, in Advances in Neural Information Processing Systems (NeurIPS), 2018, 7978–7988.
  
  Project: bigbayes tencent-lsml

  @inproceedings{NIPS2018_8021,
    author = {Chen, Jianfei and Zhu, Jun and Teh, Yee Whye and Zhang, Tong},
    booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},
    editor = {Bengio, S. and Wallach, H. and Larochelle, H. and Grauman, K. and Cesa-Bianchi, N. and Garnett, R.},
    pages = {7978--7988},
    publisher = {Curran Associates, Inc.},
    title = {Stochastic Expectation Maximization with Variance Reduction},
    year = {2018},
    month = dec,
    bdsk-url-1 = {http://papers.nips.cc/paper/8021-stochastic-expectation-maximization-with-variance-reduction.pdf}
  }
  

• H. C. L. Law , P. Zhao , J. Huang , D. Sejdinovic , Hyperparameter Learning via Distributional Transfer, ArXiv e-prints:1810.06305, 2018.
  
  Project: tencent-lsml

  Bayesian optimisation is a popular technique for hyperparameter learning but typically requires initial ’exploration’ even in cases where potentially similar prior tasks have been solved. We propose to transfer information across tasks using kernel embeddings of distributions of training datasets used in those tasks. The resulting method has a faster convergence compared to existing baselines, in some cases requiring only a few evaluations of the target objective.

  @unpublished{LawZhaHuaSej2019,
    author = {Law, Ho Chung Leon and Zhao, Peilin and Huang, Junzhou and Sejdinovic, Dino},
    title = {{{Hyperparameter Learning via Distributional Transfer}}},
    journal = {ArXiv e-prints:1810.06305},
    year = {2018}
  }