TorchOpt is a high-performance optimizer library built upon PyTorch for easy implementation of functional optimization and gradient-based meta-learning.

Overview

TorchOpt is a high-performance optimizer library built upon PyTorch for easy implementation of functional optimization and gradient-based meta-learning. It consists of two main features:

  • TorchOpt provides functional optimizer which enables JAX-like composable functional optimizer for PyTorch. With TorchOpt, one can easily conduct neural network optimization in PyTorch with functional style optimizer, similar to Optax in JAX.
  • With the desgin of functional programing, TorchOpt provides efficient, flexible, and easy-to-implement differentiable optimizer for gradient-based meta-learning research. It largely reduces the efforts required to implement sophisticated meta-learning algorithms.

The README is organized as follows:

TorchOpt as Functional Optimizer

The desgin of TorchOpt follows the philosophy of functional programming. Aligned with functorch, users can conduct functional style programing with models, optimizers and training in PyTorch. We use the Adam optimizer as an example in the following illustration. You can also check out the tutorial notebook Functional Optimizer for more details.

Optax-Like API

For those users who prefer fully functional programing, we offer Optax-Like API by passing gradients and optimizers states to the optimizer function. We design base class TorchOpt.Optimizer that has the same interface as torch.optim.Optimizer. Here is an example coupled with functorch:

import torch
from torch import nn
from torch import data
from nn import functional as F
import functorch
import TorchOpt

class Net(nn.Module):...

class Loader(data.DataLoader):...

net = Net() # init
loader = Loader()
optimizer = TorchOpt.adam()
func, params = functorch.make_functional(net)  # use functorch extract network parameters
opt_state = optimizer.init(params)  # init optimizer
xs, ys = next(loader)  # get data
pred = func(params, xs)  # forward
loss = F.cross_entropy(pred, ys)  # compute loss 
grad = torch.autograd.grad(loss, params)  # compute gradients
updates, opt_state = optimizer.update(grad, opt_state)  # get updates
params = TorchOpt.apply_updates(params, updates)  # update network parameters

PyTorch-Like API

We also offer origin PyTorch APIs (e.g. zero_grad() or step()) by warpping our Optax-Like API for traditional PyTorch user:

net = Net()  # init
loader = Loader()
optimizer = TorchOpt.Adam(net.parameters())
xs, ys = next(loader)  # get data
pred = net(xs)  # forward
loss = F.cross_entropy(pred, ys)  # compute loss
optimizer.zero_grad()  # zero gradients
loss.backward()  # backward
optimizer.step()  # step updates

Differentiable

On top of the same optimization function as torch.optim, an important benefit of functional optimizer is that one can implement differentiable optimization easily. This is particularly helpful when the algorithm requires to differentiate through optimization update (such as meta learning practices). We take as the inputs the gradients and optimizer states, use non-in-place operators to compute and output the updates. The processes can be automatically implemented, with the only need from users being to pass the argument inplace=False to the functions:

# get updates
updates, opt_state = optimizer.update(grad, opt_state, inplace=False)
# update network parameters
params = TorchOpt.apply_updates(params, updates, inplace=False)

TorchOpt as Differentiable Optimizer for Meta-Learning

Meta-Learning has gained enormous attention in both Supervised Learning and Reinforcement Learning. Meta-Learning algorithms often contain a bi-level optimisation process with inner loop updating the network parameters and outer loop updating meta parameters. The figure below illustrates the basic formulation for meta-optimization in Meta-Learning. The main feature is that the gradients of outer loss will back-propagate through all inner.step operations.

Since network parameters become a node of computation graph, a flexible Meta-Learning library should enable users manually control the gradient graph connection which means that users should have access to the network parameters and optimizer states for manually detaching or connecting the computation graph. In PyTorch designing, the network parameters or optimizer states are members of network (a.k.a. nn.Module) or optimizer (a.k.a. optim.Optimizer), this design significantly introducing difficulty for user control network parameters or optimizer states. Previous differentiable optimizer Repo higher, learn2learn follows the PyTorch designing which leads to inflexible API.

In contrast to them, TorchOpt realizes differentiable optimizer with functional programing, where Meta-Learning researchers could control the network parameters or optimizer states as normal variables (a.k.a. torch.Tensor). This functional optimizer design of TorchOpt is beneficial for implementing complex gradient flow Meta-Learning algorithms and allow us to improve computational efficiency by using techniques like operator fusion.

Meta-Learning API

  • We design a base class TorchOpt.MetaOptimizer for managing network updates in Meta-Learning. The constructor of MetaOptimizer takes as input the network rather than network parameters. MetaOptimizer exposed interface step(loss) takes as input the loss for step the network parameter. Refer to the tutorial notebook Meta Optimizer for more details.
  • We offer TorchOpt.chain which can apply a list of chainable update transformations. Combined with MetaOptimizer, it can help you conduct gradient transformation such as gradient clip before the Meta optimizer steps. Refer to the tutorial notebook Meta Optimizer for more details.
  • We observe that different Meta-Learning algorithms vary in inner-loop parameter recovery. TorchOpt provides basic functions for users to extract or recover network parameters and optimizer states anytime anywhere they want.
  • Some algorithms such as MGRL initialize the inner-loop parameters inherited from previous inner-loop process when conducting a new bi-level process. TorchOpt also provides a finer function stop_gradient for manipulating the gradient graph, which is helpful for this kind of algortihms. Refer to the notebook Stop Gradient for more details.

We give an example of MAML with inner-loop Adam optimizer to illustrate TorchOpt APIs:

net = Net() # init
# the constructor `MetaOptimizer` takes as input the network
inner_optim = TorchOpt.MetaAdam(net)
outer_optim = TorchOpt.Adam(net.parameters())

for train_iter in range(train_iters):
    outer_loss = 0
    for task in range(tasks):
        loader = Loader(tasks)
        
        # store states at the inital points
        net_state = TorchOpt.extract_state_dict(net) # extract state
        optim_state = TorchOpt.extract_state_dict(inner_optim)
        for inner_iter in range(inner_iters):
            # compute inner loss and perform inner update
            xs, ys = next(loader)
            pred = net(xs)
            inner_loss = F.cross_entropy(pred, ys)  
            inner_optim.step(inner_loss)
        # compute outer loss and back-propagate
        xs, ys = next(loader) 
        pred = net(xs)
        outer_loss += F.cross_entropy(pred, ys)
        
        # recover network and optimizer states at the inital point for the next task
        TorchOpt.recover_state_dict(inner_optim, optim_state)
        TorchOpt.recover_state_dict(net, net_state)
    
    outer_loss /= len(tasks) # task average
    outer_optim.zero_grad()
    outer_loss.backward()
    outer_optim.step()

    # stop gradient if necessary
    TorchOpt.stop_gradient(net)
    TorchOpt.stop_gradient(inner_optim)

Examples

In examples/, we offer serveral examples of functional optimizer and 5 light-weight meta-learning examples with TorchOpt. The meta-learning examples covers 2 Supervised Learning and 3 Reinforcement Learning algorithms.

High-Performance

One can think of the scale procedures on gradients of optimizer algorithms as a combination of several operations. For example, the implementation of the Adam algorithm often includes addition, multiplication, power and square operations, one can fuse these operations into several compound functions. The operator fusion could greatly simplify the computation graph and reduce the GPU function launching stall. In addition, one can also implement the optimizer backward function and manually reuse some intermediate tensors to improve the backward performance. Users can pass argument use_accelerated_op=True to adam, Adam and MetaAdam to enable the fused accelerated operator. The arguments are the same between the two kinds of implementations.

Here we evaluate the performance using the maml-omniglot code with the inner-loop Adam optimizer on GPU. We comparble the run time of the overall algorithm and the meta-optimization (outer-loop optimization) under different network architecture/inner-step numbers. We choose higher as our baseline. The figure below illustrate that our accelerated Adam can achieve at least 1/3 efficiency improvement over the baseline.

Notably, the operator fusion not only increases performance but also help simplify the computation graph, which will be discussed in the next section.

Visualization

Complex gradient flow in meta-learning brings in a great challenge for managing the gradient flow and verifying the correctness of it. TorchOpt provides a visualization tool that draw variable (e.g. network parameters or meta parameters) names on the gradient graph for better analyzing. The visualization tool is modified from torchviz. We provide an example using the visualization code. Also refer to the notebook Visualization for more details.

The figure below show the visulization result. Compared with torchviz, TorchOpt fuses the operations within the Adam together (orange) to reduce the complexity and provide simpler visualization.

Installation

Requirements

  • (Optional) For visualizing computation graphs
    • Graphviz (for Linux users use apt/yum install graphviz or conda install -c anaconda python-graphviz)
pip install TorchOpt

You can also build shared libraries from source, use:

git clone [email protected]:metaopt/TorchOpt.git
cd TorchOpt
python setup.py build_from_source

Future Plan

  • Support general implicit differentiation with functional programing.
  • Support more optimizers such as AdamW, RMSPROP
  • CPU-acclerated optimizer

The Team

TorchOpt is a work by Jie Ren, Xidong Feng, Bo Liu, Luo Mai and Yaodong Yang.

Comments
  • feat(examples): TorchRL - MAML integration

    feat(examples): TorchRL - MAML integration

    Brand-new version of the old PR

    One can test this by running

    python maml_torchrl.py --parallel --seed 1
    

    I have added the --parallel flag, to allow users to have more than one environment used.

    It can run on cuda if devices are present.

    FYI: This should run on latest main in torchrl

    enhancement example / tutorial 
    opened by vmoens 8
  • [Question] Did not manage to install developing version

    [Question] Did not manage to install developing version

    Questions

    I was trying to install the developing version of the package: I cloned the package and ran pip3 install -e as suggested in the documentation. But I obtain the following error:

     gcc -pthread -B /home/qbe/anaconda3/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/qbe/anaconda3/include/python3.7m -c flagcheck.cpp -o flagcheck.o -std=c++17
          cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
          /tmp/pip-build-env-7y5hc8wx/overlay/lib/python3.7/site-packages/setuptools/config/pyprojecttoml.py:108: _BetaConfiguration: Support for `[tool.setuptools]` in `pyproject.toml` is still *beta*.
            warnings.warn(msg, _BetaConfiguration)
          Traceback (most recent call last):
            File "/home/qbe/anaconda3/lib/python3.7/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 351, in <module>
              main()
            File "/home/qbe/anaconda3/lib/python3.7/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 333, in main
              json_out['return_val'] = hook(**hook_input['kwargs'])
            File "/home/qbe/anaconda3/lib/python3.7/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 250, in build_wheel
              metadata_directory)
            File "/tmp/pip-build-env-7y5hc8wx/overlay/lib/python3.7/site-packages/setuptools/build_meta.py", line 413, in build_wheel
              wheel_directory, config_settings)
            File "/tmp/pip-build-env-7y5hc8wx/overlay/lib/python3.7/site-packages/setuptools/build_meta.py", line 397, in _build_with_temp_dir
              self.run_setup()
            File "/tmp/pip-build-env-7y5hc8wx/overlay/lib/python3.7/site-packages/setuptools/build_meta.py", line 335, in run_setup
              exec(code, locals())
            File "<string>", line 104, in <module>
            File "/tmp/pip-build-env-7y5hc8wx/overlay/lib/python3.7/site-packages/setuptools/__init__.py", line 87, in setup
              return distutils.core.setup(**attrs)
            File "/tmp/pip-build-env-7y5hc8wx/overlay/lib/python3.7/site-packages/setuptools/_distutils/core.py", line 185, in setup
              return run_commands(dist)
            File "/tmp/pip-build-env-7y5hc8wx/overlay/lib/python3.7/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
              dist.run_commands()
            File "/tmp/pip-build-env-7y5hc8wx/overlay/lib/python3.7/site-packages/setuptools/_distutils/dist.py", line 968, in run_commands
              self.run_command(cmd)
            File "/tmp/pip-build-env-7y5hc8wx/overlay/lib/python3.7/site-packages/setuptools/dist.py", line 1217, in run_command
              super().run_command(command)
            File "/tmp/pip-build-env-7y5hc8wx/overlay/lib/python3.7/site-packages/setuptools/_distutils/dist.py", line 987, in run_command
              cmd_obj.run()
            File "/tmp/pip-build-env-7y5hc8wx/overlay/lib/python3.7/site-packages/wheel/bdist_wheel.py", line 325, in run
              self.run_command("build")
            File "/tmp/pip-build-env-7y5hc8wx/overlay/lib/python3.7/site-packages/setuptools/_distutils/cmd.py", line 319, in run_command
              self.distribution.run_command(command)
            File "/tmp/pip-build-env-7y5hc8wx/overlay/lib/python3.7/site-packages/setuptools/dist.py", line 1217, in run_command
              super().run_command(command)
            File "/tmp/pip-build-env-7y5hc8wx/overlay/lib/python3.7/site-packages/setuptools/_distutils/dist.py", line 987, in run_command
              cmd_obj.run()
            File "/tmp/pip-build-env-7y5hc8wx/overlay/lib/python3.7/site-packages/setuptools/_distutils/command/build.py", line 132, in run
              self.run_command(cmd_name)
            File "/tmp/pip-build-env-7y5hc8wx/overlay/lib/python3.7/site-packages/setuptools/_distutils/cmd.py", line 319, in run_command
              self.distribution.run_command(command)
            File "/tmp/pip-build-env-7y5hc8wx/overlay/lib/python3.7/site-packages/setuptools/dist.py", line 1217, in run_command
              super().run_command(command)
            File "/tmp/pip-build-env-7y5hc8wx/overlay/lib/python3.7/site-packages/setuptools/_distutils/dist.py", line 987, in run_command
              cmd_obj.run()
            File "/tmp/pip-build-env-7y5hc8wx/overlay/lib/python3.7/site-packages/setuptools/command/build_ext.py", line 84, in run
              _build_ext.run(self)
            File "/tmp/pip-build-env-7y5hc8wx/overlay/lib/python3.7/site-packages/setuptools/_distutils/command/build_ext.py", line 346, in run
              self.build_extensions()
            File "/tmp/pip-build-env-7y5hc8wx/overlay/lib/python3.7/site-packages/pybind11/setup_helpers.py", line 290, in build_extensions
              super().build_extensions()
            File "/tmp/pip-build-env-7y5hc8wx/overlay/lib/python3.7/site-packages/setuptools/_distutils/command/build_ext.py", line 466, in build_extensions
              self._build_extensions_serial()
            File "/tmp/pip-build-env-7y5hc8wx/overlay/lib/python3.7/site-packages/setuptools/_distutils/command/build_ext.py", line 492, in _build_extensions_serial
              self.build_extension(ext)
            File "<string>", line 42, in build_extension
          RuntimeError: Cannot find CMake executable.
          [end of output]
      
      note: This error originates from a subprocess, and is likely not a problem with pip.
      ERROR: Failed building wheel for torchopt
    Failed to build torchopt
    ERROR: Could not build wheels for torchopt, which is required to install pyproject.toml-based projects 
    

    Checklist

    • [x] I have checked that there is no similar issue in the repo. (required)
    • [x] I have read the documentation. (required)
    question 
    opened by QB3 5
  • [BUG] metaopt/optree is not a public repo

    [BUG] metaopt/optree is not a public repo

    Describe the bug

    I have trouble installing optree from pypi on our compute clusters, getting undefined symbol errors. I saw this link to an optree github repo on pypi, so I thought I can compile it from source, but the repo either doesn't exist or is not public.

    To Reproduce

    Visit https://github.com/metaopt/optree without logging into a collaborator account.

    Expected behavior

    The repo should be accessible

    System info

    N/A

    Additional context

    Our compute cluster uses a module system with customized builds, sometimes requiring us to build packages from source.

    Checklist

    • [x] I have checked that there is no similar issue in the repo (required)
    • [x] I have read the documentation (required)
    • [x] I have provided a minimal working example to reproduce the bug (required)
    bug 
    opened by vmichals 4
  • [BUG] Error in tutorials/3_Meta_Optimizer.ipynb

    [BUG] Error in tutorials/3_Meta_Optimizer.ipynb

    Describe the bug

    Error in running tutorials/3_Meta_Optimizer.ipynb cell [6] (the first cell under section 2.1 Basic API)

    ---------------------------------------------------------------------------
    RuntimeError                              Traceback (most recent call last)
    [<ipython-input-9-ad332932d4e7>](https://localhost:8080/#) in <module>
         15 for i in range(2):
         16     inner_loss = net(x)
    ---> 17     optim.step(inner_loss)
         18 
         19 print(f'a = {net.a!r}')
    
    6 frames
    [/usr/local/lib/python3.7/dist-packages/torchopt/_src/optimizer/meta/base.py](https://localhost:8080/#) in step(self, loss)
         68                 new_state,
         69                 params=flattened_params,
    ---> 70                 inplace=False,
         71             )
         72             self.state_groups[i] = new_state
    
    [/usr/local/lib/python3.7/dist-packages/torchopt/_src/transform.py](https://localhost:8080/#) in update_fn(updates, state, params, inplace)
         65 
         66         flattened_updates, state = inner.update(
    ---> 67             flattened_updates, state, params=params, inplace=inplace
         68         )
         69         updates = pytree.tree_unflatten(treedef, flattened_updates)
    
    [/usr/local/lib/python3.7/dist-packages/torchopt/_src/base.py](https://localhost:8080/#) in update_fn(updates, state, params, inplace)
        181             new_state = []
        182             for s, fn in zip(state, update_fns):  # pylint: disable=invalid-name
    --> 183                 updates, new_s = fn(updates, s, params=params, inplace=inplace)
        184                 new_state.append(new_s)
        185             return updates, tuple(new_state)
    
    [/usr/local/lib/python3.7/dist-packages/torchopt/_src/transform.py](https://localhost:8080/#) in update_fn(updates, state, params, inplace)
        312     def update_fn(updates, state, *, params=None, inplace=True):  # pylint: disable=unused-argument
        313         mu = _update_moment(
    --> 314             updates, state.mu, b1, order=1, inplace=inplace, already_flattened=already_flattened
        315         )
        316         nu = _update_moment(
    
    [/usr/local/lib/python3.7/dist-packages/torchopt/_src/transform.py](https://localhost:8080/#) in _update_moment(updates, moments, decay, order, inplace, already_flattened)
        214 
        215     if already_flattened:
    --> 216         return map_flattened(f, updates, moments)
        217     return pytree.tree_map(f, updates, moments)
        218 
    
    [/usr/local/lib/python3.7/dist-packages/torchopt/_src/transform.py](https://localhost:8080/#) in map_flattened(func, *args)
         49 def map_flattened(func: Callable, *args: Any) -> List[Any]:
         50     """Apply a function to each element of a flattened list."""
    ---> 51     return list(map(func, *args))
         52 
         53 
    
    [/usr/local/lib/python3.7/dist-packages/torchopt/_src/transform.py](https://localhost:8080/#) in f(g, t)
        211 
        212             def f(g, t):
    --> 213                 return t.mul(decay).add_(g, alpha=1 - decay) if g is not None else t
        214 
        215     if already_flattened:
    
    RuntimeError: output with shape [] doesn't match the broadcast shape [1]
    

    To Reproduce

    See the following notebook for detailed steps to reproduce https://colab.research.google.com/drive/14S7DzcovrUwDkZuRxdq5OW5KwYqWTJsy?usp=sharing

    Expected behavior

    Screenshots

    System info

    Describe the characteristic of your environment:

    • Describe how the library was installed (pip, source, ...)
    • Python version
    • Versions of any other relevant libraries
    import torchopt, numpy, sys
    print(torchopt.__version__, numpy.__version__, sys.version, sys.platform)
    
    0.5.0 1.21.6 3.7.13 (default, Apr 24 2022, 01:04:09) 
    [GCC 7.5.0] linux
    

    Additional context

    Add any other context about the problem here.

    Reason and Possible fixes

    If you know or suspect the reason for this bug, paste the code lines and suggest modifications.

    Checklist

    • [x] I have checked that there is no similar issue in the repo (required)
    • [x] I have read the documentation (required)
    • [x] I have provided a minimal working example to reproduce the bug (required)
    bug 
    opened by xianghang 3
  • [BUG] module 'optree' has no attribute 'PyTreeDef'

    [BUG] module 'optree' has no attribute 'PyTreeDef'

    Describe the bug

    I used torchopt in google colab without any problem untill last week, an error occurred when I tried to import torchopt.

    AttributeError: module 'optree' has no attribute 'PyTreeDef'
    

    I have tried reinstalling torchop as well as pytorch, and even replacing a local machine to execute the program, but the problem has not been solved.

    To Reproduce

    The error occurs when executing the following code on Google Colab:

    !pip3 install torchopt
    import torchopt
    

    Then

    ---------------------------------------------------------------------------
    AttributeError                            Traceback (most recent call last)
    Cell In [2], line 1
    ----> 1 import torchopt
    
    File ~/.pyenv/versions/anaconda3-5.3.1/envs/meta_rl/lib/python3.8/site-packages/torchopt/__init__.py:17
          1 # Copyright 2022 MetaOPT Team. All Rights Reserved.
          2 #
          3 # Licensed under the Apache License, Version 2.0 (the "License");
       (...)
         13 # limitations under the License.
         14 # ==============================================================================
         15 """TorchOpt: a high-performance optimizer library built upon PyTorch."""
    ---> 17 from torchopt._src import (
         18     accelerated_op_available,
         19     clip,
         20     combine,
         21     hook,
         22     implicit_diff,
         23     linear_solve,
         24     schedule,
         25     visual,
         26 )
         27 from torchopt._src.alias import adam, adamw, rmsprop, sgd
         28 from torchopt._src.clip import clip_grad_norm
    
    File ~/.pyenv/versions/anaconda3-5.3.1/envs/meta_rl/lib/python3.8/site-packages/torchopt/_src/implicit_diff.py:159
        153     out_kwargs = {k: v for k, (_, _, v) in bound.kwargs.items()}
        154     return out_args, out_kwargs, map_args_back
        157 def _split_tensor_and_others(
        158     mixed_tuple: Tuple,
    --> 159 ) -> Tuple[pytree.PyTreeDef, Tuple[bool, ...], Tuple[torch.Tensor, ...], Tuple[Any, ...]]:
        160     flattened, treedef = pytree.tree_flatten(mixed_tuple)
        161     tensors = []
    
    AttributeError: module 'optree' has no attribute 'PyTreeDef'
    

    System info

    • CUDA Version: 11.0
    • python 3.8
    • torch 1.13.0

    Checklist

    • [x] I have checked that there is no similar issue in the repo. (required)
    • [x] I have read the documentation. (required)
    • [x] I have provided a minimal working example to reproduce the bug. (required)
    bug dependencies 
    opened by woithook 2
  • feat(linear_solve): matrix inversion linear solver with neumann series approximation

    feat(linear_solve): matrix inversion linear solver with neumann series approximation

    Description

    matrix inversion linear solver with neumann series approximation

    Types of changes

    • [x] New feature (non-breaking change which adds core functionality)
    • [x] Documentation (update in the documentation)
    • [x] Example (update in the folder of example)

    Checklist

    • [x] I have read the CONTRIBUTION guide (required)
    • [x] My change requires a change to the documentation.
    • [x] I have updated the tests accordingly (required for a bug fix or a new feature).
    • [x] I have updated the documentation accordingly.
    • [x] I have reformatted the code using make format (required)
    • [x] I have checked the code using make lint (required)
    • [x] I have ensured make test pass. (required)
    enhancement feature 
    opened by Benjamin-eecs 2
  • pip prod(deps): update doc8 requirement from <1.0.0a0 to <2.0.0

    pip prod(deps): update doc8 requirement from <1.0.0a0 to <2.0.0

    Updates the requirements on doc8 to permit the latest version.

    Release notes

    Sourced from doc8's releases.

    v1.0.0

    Major Changes

    Bugfixes

    Commits

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    dependencies 
    opened by dependabot[bot] 2
  • fix(CMakeLists.txt): fix CUDA build for accelerated OP

    fix(CMakeLists.txt): fix CUDA build for accelerated OP

    Description

    Describe your changes in detail.

    Motivation and Context

    Why is this change required? What problem does it solve? If it fixes an open issue, please link to the issue here. You can use the syntax close #15213 if this solves the issue #15213

    • [X] I have raised an issue to propose this change (required for new features and bug fixes)

    Fix CUDA build by manually adding a new macro __CUDA_ENABLED__. Macro __CUDACC__ is not always defined by nvcc. See:

    https://github.com/pytorch/pytorch/blob/4d3f7df33fa49a80b20ae328d5a5d71c655e3634/cmake/Modules_CUDA_fix/upstream/FindCUDA/run_nvcc.cmake#L190-L194

    image

    Types of changes

    What types of changes does your code introduce? Put an x in all the boxes that apply:

    • [X] Bug fix (non-breaking change which fixes an issue)
    • [ ] New feature (non-breaking change which adds core functionality)
    • [ ] New environment (non-breaking change which adds 3rd-party environment)
    • [ ] Breaking change (fix or feature that would cause existing functionality to change)
    • [ ] Documentation (update in the documentation)
    • [ ] Example (update in the folder of example)

    Checklist

    Go over all the following points, and put an x in all the boxes that apply. If you are unsure about any of these, don't hesitate to ask. We are here to help!

    • [X] I have read the CONTRIBUTION guide (required)
    • [X] My change requires a change to the documentation.
    • [X] I have updated the tests accordingly (required for a bug fix or a new feature).
    • [X] I have updated the documentation accordingly.
    • [X] I have reformatted the code using make format (required)
    • [X] I have checked the code using make lint (required)
    • [X] I have ensured make test pass. (required)
    bug cxx / cuda 
    opened by XuehaiPan 2
  • [BUG] Separate CPU / CUDA wheels

    [BUG] Separate CPU / CUDA wheels

    Describe the bug

    A clear and concise description of what the bug is.

    Wheels built with torch==1.12.0+cu116 is incompatible with torch==1.12.0+cpu (see CI output https://github.com/metaopt/TorchOpt/runs/7553102052 for more details). Different torch build ships with different libraries:

    torch==1.12.0+cu116:

    $ ls $SITE_PACKAGES/torch/lib
    total 3.3G
    -rwxr-xr-x 1 root root 1.2M Jul 28 04:01 libc10_cuda.so
    -rwxr-xr-x 1 root root 751K Jul 28 04:01 libc10.so
    -rwxr-xr-x 1 root root  25K Jul 28 04:01 libcaffe2_nvrtc.so
    -rwxr-xr-x 1 root root 335M Jul 28 04:01 libcublasLt.so.11
    -rwxr-xr-x 1 root root 150M Jul 28 04:01 libcublas.so.11
    -rwxr-xr-x 1 root root 668K Jul 28 04:01 libcudart-45da57e3.so.11.0
    -rwxr-xr-x 1 root root 124M Jul 28 04:01 libcudnn_adv_infer.so.8
    -rwxr-xr-x 1 root root  92M Jul 28 04:01 libcudnn_adv_train.so.8
    -rwxr-xr-x 1 root root 774M Jul 28 04:01 libcudnn_cnn_infer.so.8
    -rwxr-xr-x 1 root root  85M Jul 28 04:01 libcudnn_cnn_train.so.8
    -rwxr-xr-x 1 root root  86M Jul 28 04:01 libcudnn_ops_infer.so.8
    -rwxr-xr-x 1 root root  68M Jul 28 04:01 libcudnn_ops_train.so.8
    -rwxr-xr-x 1 root root 155K Jul 28 04:01 libcudnn.so.8
    -rwxr-xr-x 1 root root 165K Jul 28 04:01 libgomp-a34b3233.so.1
    -rwxr-xr-x 1 root root  44M Jul 28 04:01 libnvrtc-4dd39364.so.11.2
    -rwxr-xr-x 1 root root 6.8M Jul 28 04:01 libnvrtc-builtins.so.11.6
    -rwxr-xr-x 1 root root  43K Jul 28 04:01 libnvToolsExt-847d78f2.so.1
    -rwxr-xr-x 1 root root  44K Jul 28 04:01 libshm.so
    -rwxr-xr-x 1 root root 487M Jul 28 04:01 libtorch_cpu.so
    -rwxr-xr-x 1 root root 216M Jul 28 04:01 libtorch_cuda_cpp.so
    -rwxr-xr-x 1 root root 653M Jul 28 04:01 libtorch_cuda_cu.so
    -rwxr-xr-x 1 root root 209M Jul 28 04:01 libtorch_cuda_linalg.so
    -rwxr-xr-x 1 root root 163K Jul 28 04:01 libtorch_cuda.so
    -rwxr-xr-x 1 root root  21K Jul 28 04:01 libtorch_global_deps.so
    -rwxr-xr-x 1 root root  21M Jul 28 04:01 libtorch_python.so
    -rwxr-xr-x 1 root root  16K Jul 28 04:01 libtorch.so
    

    torch==1.12.0+cpu:

    $ ls $SITE_PACKAGES/torch/lib
    total 496M
    -rwxr-xr-x 1 root root 269K Jul 28 04:02 libbackend_with_compiler.so
    -rwxr-xr-x 1 root root 766K Jul 28 04:02 libc10.so
    -rwxr-xr-x 1 root root 165K Jul 28 04:02 libgomp-a34b3233.so.1
    -rwxr-xr-x 1 root root 228K Jul 28 04:02 libjitbackend_test.so
    -rwxr-xr-x 1 root root  35K Jul 28 04:02 libshm.so
    -rwxr-xr-x 1 root root 588K Jul 28 04:02 libtorchbind_test.so
    -rwxr-xr-x 1 root root 476M Jul 28 04:02 libtorch_cpu.so
    -rwxr-xr-x 1 root root 8.6K Jul 28 04:02 libtorch_global_deps.so
    -rwxr-xr-x 1 root root  19M Jul 28 04:02 libtorch_python.so
    -rwxr-xr-x 1 root root 7.1K Jul 28 04:02 libtorch.so
    

    In our .cxx and .cu code, we only have one include directive #include <torch/extension.h> and only referenced torch::Tensor and AT_DISPATCH_FLOATING_TYPES. But the built shared libraries are linking against too many libraries than expected.

    $ ldd /tmp/tmp.3M50Q7bV1d/venv/lib/python3.7/site-packages/torchopt/_lib/adam_op.cpython-37m-x86_64-linux-gnu.so
            linux-vdso.so.1 =>  (0x00007ffcd44ea000)
            libc10.so => /tmp/tmp.3M50Q7bV1d/venv/lib/python3.7/site-packages/torchopt/_lib/../../torch/lib/libc10.so (0x00007f75c1243000)
            libc10_cuda.so => /tmp/tmp.3M50Q7bV1d/venv/lib/python3.7/site-packages/torchopt/_lib/../../torch/lib/libc10_cuda.so (0x00007f75c109b000)
            libcaffe2_nvrtc.so => /tmp/tmp.3M50Q7bV1d/venv/lib/python3.7/site-packages/torchopt/_lib/../../torch/lib/libcaffe2_nvrtc.so (0x00007f75c123c000)
            libshm.so => /tmp/tmp.3M50Q7bV1d/venv/lib/python3.7/site-packages/torchopt/_lib/../../torch/lib/libshm.so (0x00007f75c1231000)
            libtorch.so => /tmp/tmp.3M50Q7bV1d/venv/lib/python3.7/site-packages/torchopt/_lib/../../torch/lib/libtorch.so (0x00007f75c122c000)
            libtorch_cpu.so => /tmp/tmp.3M50Q7bV1d/venv/lib/python3.7/site-packages/torchopt/_lib/../../torch/lib/libtorch_cpu.so (0x00007f75a704d000)
            libtorch_cuda.so => /tmp/tmp.3M50Q7bV1d/venv/lib/python3.7/site-packages/torchopt/_lib/../../torch/lib/libtorch_cuda.so (0x00007f75c120c000)
            libtorch_cuda_cpp.so => /tmp/tmp.3M50Q7bV1d/venv/lib/python3.7/site-packages/torchopt/_lib/../../torch/lib/libtorch_cuda_cpp.so (0x00007f7599db0000)
            libtorch_cuda_cu.so => /tmp/tmp.3M50Q7bV1d/venv/lib/python3.7/site-packages/torchopt/_lib/../../torch/lib/libtorch_cuda_cu.so (0x00007f757237b000)
            libtorch_cuda_linalg.so => /tmp/tmp.3M50Q7bV1d/venv/lib/python3.7/site-packages/torchopt/_lib/../../torch/lib/libtorch_cuda_linalg.so (0x00007f7565905000)
            libtorch_global_deps.so => /tmp/tmp.3M50Q7bV1d/venv/lib/python3.7/site-packages/torchopt/_lib/../../torch/lib/libtorch_global_deps.so (0x00007f75c1203000)
            libtorch_python.so => /tmp/tmp.3M50Q7bV1d/venv/lib/python3.7/site-packages/torchopt/_lib/../../torch/lib/libtorch_python.so (0x00007f7564957000)
            librt.so.1 => /lib64/librt.so.1 (0x00007f756474f000)
            libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f7564533000)
            libdl.so.2 => /lib64/libdl.so.2 (0x00007f756432f000)
            libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007f7564027000)
            libm.so.6 => /lib64/libm.so.6 (0x00007f7563d25000)
            libgomp.so.1 => /lib64/libgomp.so.1 (0x00007f7563aff000)
            libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f75638e9000)
            libc.so.6 => /lib64/libc.so.6 (0x00007f756351b000)
            /lib64/ld-linux-x86-64.so.2 (0x00007f75c1199000)
            libgomp-a34b3233.so.1 => /tmp/tmp.3M50Q7bV1d/venv/lib/python3.7/site-packages/torchopt/_lib/../../torch/lib/libgomp-a34b3233.so.1 (0x00007f75632f1000)
            libcuda.so.1 => /lib64/libcuda.so.1 (0x00007f7561e96000)
            libnvrtc-4dd39364.so.11.2 => /tmp/tmp.3M50Q7bV1d/venv/lib/python3.7/site-packages/torchopt/_lib/../../torch/lib/libnvrtc-4dd39364.so.11.2 (0x00007f755f075000)
            libcudart-45da57e3.so.11.0 => /tmp/tmp.3M50Q7bV1d/venv/lib/python3.7/site-packages/torchopt/_lib/../../torch/lib/libcudart-45da57e3.so.11.0 (0x00007f755edcd000)
            libnvToolsExt-847d78f2.so.1 => /tmp/tmp.3M50Q7bV1d/venv/lib/python3.7/site-packages/torchopt/_lib/../../torch/lib/libnvToolsExt-847d78f2.so.1 (0x00007f755ebc2000)
            libcudnn.so.8 => /tmp/tmp.3M50Q7bV1d/venv/lib/python3.7/site-packages/torchopt/_lib/../../torch/lib/libcudnn.so.8 (0x00007f755e99a000)
            libcublas.so.11 => /tmp/tmp.3M50Q7bV1d/venv/lib/python3.7/site-packages/torchopt/_lib/../../torch/lib/libcublas.so.11 (0x00007f755521c000)
            libcublasLt.so.11 => /tmp/tmp.3M50Q7bV1d/venv/lib/python3.7/site-packages/torchopt/_lib/../../torch/lib/libcublasLt.so.11 (0x00007f75401b6000)
    

    To Reproduce

    Steps to reproduce the behavior.

    Please try to provide a minimal example to reproduce the bug. Error messages and stack traces are also helpful.

    Please use the markdown code blocks for both code and stack traces.

    See CI output https://github.com/metaopt/TorchOpt/runs/7553102052 for more details.

    $ git clone [email protected]:XuehaiPan/TorchOpt.git && cd TorchOpt
    $ git checkout cibuildwheel
    $ pip3 install --upgrade cibuildwheel
    $ PIP_EXTRA_INDEX_URL="https://download.pytorch.org/whl/cu116" python3 -m cibuildwheel --platform linux --config-file=pyproject.toml
    ...
    ls /tmp/tmp.3M50Q7bV1d/venv/lib/python3.7/site-packages/torch/lib
    total 496M
    -rwxr-xr-x 1 root root 269K Jul 28 04:02 libbackend_with_compiler.so
    -rwxr-xr-x 1 root root 766K Jul 28 04:02 libc10.so
    -rwxr-xr-x 1 root root 165K Jul 28 04:02 libgomp-a34b3233.so.1
    -rwxr-xr-x 1 root root 228K Jul 28 04:02 libjitbackend_test.so
    -rwxr-xr-x 1 root root  35K Jul 28 04:02 libshm.so
    -rwxr-xr-x 1 root root 588K Jul 28 04:02 libtorchbind_test.so
    -rwxr-xr-x 1 root root 476M Jul 28 04:02 libtorch_cpu.so
    -rwxr-xr-x 1 root root 8.6K Jul 28 04:02 libtorch_global_deps.so
    -rwxr-xr-x 1 root root  19M Jul 28 04:02 libtorch_python.so
    -rwxr-xr-x 1 root root 7.1K Jul 28 04:02 libtorch.so
    ldd /tmp/tmp.3M50Q7bV1d/venv/lib/python3.7/site-packages/torchopt/_lib/adam_op.cpython-37m-x86_64-linux-gnu.so
            linux-vdso.so.1 =>  (0x00007ffd5997a000)
            libc10.so => /tmp/tmp.3M50Q7bV1d/venv/lib/python3.7/site-packages/torchopt/_lib/../../torch/lib/libc10.so (0x00007ff16efde000)
            libc10_cuda.so => not found
            libcaffe2_nvrtc.so => not found
            libshm.so => /tmp/tmp.3M50Q7bV1d/venv/lib/python3.7/site-packages/torchopt/_lib/../../torch/lib/libshm.so (0x00007ff16efce000)
            libtorch.so => /tmp/tmp.3M50Q7bV1d/venv/lib/python3.7/site-packages/torchopt/_lib/../../torch/lib/libtorch.so (0x00007ff16efcb000)
            libtorch_cpu.so => /tmp/tmp.3M50Q7bV1d/venv/lib/python3.7/site-packages/torchopt/_lib/../../torch/lib/libtorch_cpu.so (0x00007ff155b9a000)
            libtorch_cuda.so => not found
            libtorch_cuda_cpp.so => not found
            libtorch_cuda_cu.so => not found
            libtorch_cuda_linalg.so => not found
            libtorch_global_deps.so => /tmp/tmp.3M50Q7bV1d/venv/lib/python3.7/site-packages/torchopt/_lib/../../torch/lib/libtorch_global_deps.so (0x00007ff16efc5000)
            libtorch_python.so => /tmp/tmp.3M50Q7bV1d/venv/lib/python3.7/site-packages/torchopt/_lib/../../torch/lib/libtorch_python.so (0x00007ff154dc0000)
            librt.so.1 => /lib64/librt.so.1 (0x00007ff154bb8000)
            libpthread.so.0 => /lib64/libpthread.so.0 (0x00007ff15499c000)
            libdl.so.2 => /lib64/libdl.so.2 (0x00007ff154798000)
            libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007ff154490000)
            libm.so.6 => /lib64/libm.so.6 (0x00007ff15418e000)
            libgomp.so.1 => /lib64/libgomp.so.1 (0x00007ff153f68000)
            libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007ff153d52000)
            libc.so.6 => /lib64/libc.so.6 (0x00007ff153984000)
            /lib64/ld-linux-x86-64.so.2 (0x00007ff16ef39000)
            libgomp-a34b3233.so.1 => /tmp/tmp.3M50Q7bV1d/venv/lib/python3.7/site-packages/torchopt/_lib/../../torch/lib/libgomp-a34b3233.so.1 (0x00007ff15375a000)
    patchelf --print-rpath /tmp/tmp.3M50Q7bV1d/venv/lib/python3.7/site-packages/torchopt/_lib/adam_op.cpython-37m-x86_64-linux-gnu.so
    $ORIGIN/../../torch/lib:$ORIGIN/../../torchopt.libs
    make: Entering directory `/project'
    /tmp/tmp.3M50Q7bV1d/venv/bin/python3 -m pip show pytest &>/dev/null || (cd && /tmp/tmp.3M50Q7bV1d/venv/bin/python3 -m pip install pytest --upgrade)
    /tmp/tmp.3M50Q7bV1d/venv/bin/python3 -m pip show pytest_cov &>/dev/null || (cd && /tmp/tmp.3M50Q7bV1d/venv/bin/python3 -m pip install pytest_cov --upgrade)
    /tmp/tmp.3M50Q7bV1d/venv/bin/python3 -m pip show pytest_xdist &>/dev/null || (cd && /tmp/tmp.3M50Q7bV1d/venv/bin/python3 -m pip install pytest_xdist --upgrade)
    cd tests && /tmp/tmp.3M50Q7bV1d/venv/bin/python3 -m pytest unit --cov torchopt --durations 0 -v --cov-report term-missing --color=yes
    ============================= test session starts ==============================
    platform linux -- Python 3.7.13, pytest-7.1.2, pluggy-1.0.0 -- /tmp/tmp.3M50Q7bV1d/venv/bin/python3
    cachedir: .pytest_cache
    rootdir: /project
    plugins: forked-1.4.0, cov-3.0.0, xdist-2.5.0
    collecting ... collected 0 items / 4 errors
    
    ==================================== ERRORS ====================================
    ___________________ ERROR collecting tests/unit/test_clip.py ___________________
    ImportError while importing test module '/project/tests/unit/test_clip.py'.
    Hint: make sure your test modules/packages have valid Python names.
    Traceback:
    /opt/python/cp37-cp37m/lib/python3.7/importlib/__init__.py:127: in import_module
        return _bootstrap._gcd_import(name[level:], package, level)
    /workspace/tests/unit/test_clip.py:25: in <module>
        ???
    /tmp/tmp.3M50Q7bV1d/venv/lib/python3.7/site-packages/torchopt/__init__.py:17: in <module>
        from torchopt._src import accelerated_op_available, clip, combine, hook, schedule, visual
    /tmp/tmp.3M50Q7bV1d/venv/lib/python3.7/site-packages/torchopt/_src/__init__.py:16: in <module>
        from torchopt._src.accelerated_op import accelerated_op_available
    /tmp/tmp.3M50Q7bV1d/venv/lib/python3.7/site-packages/torchopt/_src/accelerated_op/__init__.py:20: in <module>
        from torchopt._src.accelerated_op.adam_op import AdamOp
    /tmp/tmp.3M50Q7bV1d/venv/lib/python3.7/site-packages/torchopt/_src/accelerated_op/adam_op/__init__.py:16: in <module>
        from torchopt._src.accelerated_op.adam_op.adam_op import AdamOp
    /tmp/tmp.3M50Q7bV1d/venv/lib/python3.7/site-packages/torchopt/_src/accelerated_op/adam_op/adam_op.py:22: in <module>
        from torchopt._lib import adam_op  # pylint: disable=no-name-in-module
    E   ImportError: libc10_cuda.so: cannot open shared object file: No such file or directory
    _________________ ERROR collecting tests/unit/test_schedule.py _________________
    ImportError while importing test module '/project/tests/unit/test_schedule.py'.
    Hint: make sure your test modules/packages have valid Python names.
    Traceback:
    /opt/python/cp37-cp37m/lib/python3.7/importlib/__init__.py:127: in import_module
        return _bootstrap._gcd_import(name[level:], package, level)
    /workspace/tests/unit/test_schedule.py:18: in <module>
        ???
    /tmp/tmp.3M50Q7bV1d/venv/lib/python3.7/site-packages/torchopt/__init__.py:17: in <module>
        from torchopt._src import accelerated_op_available, clip, combine, hook, schedule, visual
    /tmp/tmp.3M50Q7bV1d/venv/lib/python3.7/site-packages/torchopt/_src/__init__.py:16: in <module>
        from torchopt._src.accelerated_op import accelerated_op_available
    /tmp/tmp.3M50Q7bV1d/venv/lib/python3.7/site-packages/torchopt/_src/accelerated_op/__init__.py:20: in <module>
        from torchopt._src.accelerated_op.adam_op import AdamOp
    /tmp/tmp.3M50Q7bV1d/venv/lib/python3.7/site-packages/torchopt/_src/accelerated_op/adam_op/__init__.py:16: in <module>
        from torchopt._src.accelerated_op.adam_op.adam_op import AdamOp
    /tmp/tmp.3M50Q7bV1d/venv/lib/python3.7/site-packages/torchopt/_src/accelerated_op/adam_op/adam_op.py:22: in <module>
        from torchopt._lib import adam_op  # pylint: disable=no-name-in-module
    E   ImportError: libc10_cuda.so: cannot open shared object file: No such file or directory
    ______ ERROR collecting tests/unit/high_level/test_high_level_inplace.py _______
    ImportError while importing test module '/project/tests/unit/high_level/test_high_level_inplace.py'.
    Hint: make sure your test modules/packages have valid Python names.
    Traceback:
    /opt/python/cp37-cp37m/lib/python3.7/importlib/__init__.py:127: in import_module
        return _bootstrap._gcd_import(name[level:], package, level)
    /workspace/tests/unit/high_level/test_high_level_inplace.py:25: in <module>
        ???
    /tmp/tmp.3M50Q7bV1d/venv/lib/python3.7/site-packages/torchopt/__init__.py:17: in <module>
        from torchopt._src import accelerated_op_available, clip, combine, hook, schedule, visual
    /tmp/tmp.3M50Q7bV1d/venv/lib/python3.7/site-packages/torchopt/_src/__init__.py:16: in <module>
        from torchopt._src.accelerated_op import accelerated_op_available
    /tmp/tmp.3M50Q7bV1d/venv/lib/python3.7/site-packages/torchopt/_src/accelerated_op/__init__.py:20: in <module>
        from torchopt._src.accelerated_op.adam_op import AdamOp
    /tmp/tmp.3M50Q7bV1d/venv/lib/python3.7/site-packages/torchopt/_src/accelerated_op/adam_op/__init__.py:16: in <module>
        from torchopt._src.accelerated_op.adam_op.adam_op import AdamOp
    /tmp/tmp.3M50Q7bV1d/venv/lib/python3.7/site-packages/torchopt/_src/accelerated_op/adam_op/adam_op.py:22: in <module>
        from torchopt._lib import adam_op  # pylint: disable=no-name-in-module
    E   ImportError: libc10_cuda.so: cannot open shared object file: No such file or directory
    _______ ERROR collecting tests/unit/low_level/test_low_level_inplace.py ________
    ImportError while importing test module '/project/tests/unit/low_level/test_low_level_inplace.py'.
    Hint: make sure your test modules/packages have valid Python names.
    Traceback:
    /opt/python/cp37-cp37m/lib/python3.7/importlib/__init__.py:127: in import_module
        return _bootstrap._gcd_import(name[level:], package, level)
    /workspace/tests/unit/low_level/test_low_level_inplace.py:26: in <module>
        ???
    /tmp/tmp.3M50Q7bV1d/venv/lib/python3.7/site-packages/torchopt/__init__.py:17: in <module>
        from torchopt._src import accelerated_op_available, clip, combine, hook, schedule, visual
    /tmp/tmp.3M50Q7bV1d/venv/lib/python3.7/site-packages/torchopt/_src/__init__.py:16: in <module>
        from torchopt._src.accelerated_op import accelerated_op_available
    /tmp/tmp.3M50Q7bV1d/venv/lib/python3.7/site-packages/torchopt/_src/accelerated_op/__init__.py:20: in <module>
        from torchopt._src.accelerated_op.adam_op import AdamOp
    /tmp/tmp.3M50Q7bV1d/venv/lib/python3.7/site-packages/torchopt/_src/accelerated_op/adam_op/__init__.py:16: in <module>
        from torchopt._src.accelerated_op.adam_op.adam_op import AdamOp
    /tmp/tmp.3M50Q7bV1d/venv/lib/python3.7/site-packages/torchopt/_src/accelerated_op/adam_op/adam_op.py:22: in <module>
        from torchopt._lib import adam_op  # pylint: disable=no-name-in-module
    E   ImportError: libc10_cuda.so: cannot open shared object file: No such file or directory
    
    ---------- coverage: platform linux, python 3.7.13-final-0 -----------
    Name                                                                                                    Stmts   Miss  Cover   Missing
    -------------------------------------------------------------------------------------------------------------------------------------
    /tmp/tmp.3M50Q7bV1d/venv/lib/python3.7/site-packages/torchopt/__init__.py                                   9      7    22%   18-26
    /tmp/tmp.3M50Q7bV1d/venv/lib/python3.7/site-packages/torchopt/_lib/__init__.py                              0      0   100%
    /tmp/tmp.3M50Q7bV1d/venv/lib/python3.7/site-packages/torchopt/_src/__init__.py                              1      0   100%
    /tmp/tmp.3M50Q7bV1d/venv/lib/python3.7/site-packages/torchopt/_src/accelerated_op/__init__.py              21     18    14%   23-45
    /tmp/tmp.3M50Q7bV1d/venv/lib/python3.7/site-packages/torchopt/_src/accelerated_op/adam_op/__init__.py       1      0   100%
    /tmp/tmp.3M50Q7bV1d/venv/lib/python3.7/site-packages/torchopt/_src/accelerated_op/adam_op/adam_op.py       75     72     4%   25-137
    make: *** [pytest] Error 2
    -------------------------------------------------------------------------------------------------------------------------------------
    TOTAL                                                                                                     107     97     9%
    
    =========================== short test summary info ============================
    ERROR unit/test_clip.py
    ERROR unit/test_schedule.py
    ERROR unit/high_level/test_high_level_inplace.py
    ERROR unit/low_level/test_low_level_inplace.py
    !!!!!!!!!!!!!!!!!!! Interrupted: 4 errors during collection !!!!!!!!!!!!!!!!!!!!
    ============================== 4 errors in 0.82s ===============================
    ...
    

    Shared library compiled with torch==1.12.0+cu116:

    $ ls /tmp/tmp.3M50Q7bV1d/venv/lib/python3.7/site-packages/torch/lib
    total 3.3G
    -rwxr-xr-x 1 root root 1.2M Jul 28 04:01 libc10_cuda.so
    -rwxr-xr-x 1 root root 751K Jul 28 04:01 libc10.so
    -rwxr-xr-x 1 root root  25K Jul 28 04:01 libcaffe2_nvrtc.so
    -rwxr-xr-x 1 root root 335M Jul 28 04:01 libcublasLt.so.11
    -rwxr-xr-x 1 root root 150M Jul 28 04:01 libcublas.so.11
    -rwxr-xr-x 1 root root 668K Jul 28 04:01 libcudart-45da57e3.so.11.0
    -rwxr-xr-x 1 root root 124M Jul 28 04:01 libcudnn_adv_infer.so.8
    -rwxr-xr-x 1 root root  92M Jul 28 04:01 libcudnn_adv_train.so.8
    -rwxr-xr-x 1 root root 774M Jul 28 04:01 libcudnn_cnn_infer.so.8
    -rwxr-xr-x 1 root root  85M Jul 28 04:01 libcudnn_cnn_train.so.8
    -rwxr-xr-x 1 root root  86M Jul 28 04:01 libcudnn_ops_infer.so.8
    -rwxr-xr-x 1 root root  68M Jul 28 04:01 libcudnn_ops_train.so.8
    -rwxr-xr-x 1 root root 155K Jul 28 04:01 libcudnn.so.8
    -rwxr-xr-x 1 root root 165K Jul 28 04:01 libgomp-a34b3233.so.1
    -rwxr-xr-x 1 root root  44M Jul 28 04:01 libnvrtc-4dd39364.so.11.2
    -rwxr-xr-x 1 root root 6.8M Jul 28 04:01 libnvrtc-builtins.so.11.6
    -rwxr-xr-x 1 root root  43K Jul 28 04:01 libnvToolsExt-847d78f2.so.1
    -rwxr-xr-x 1 root root  44K Jul 28 04:01 libshm.so
    -rwxr-xr-x 1 root root 487M Jul 28 04:01 libtorch_cpu.so
    -rwxr-xr-x 1 root root 216M Jul 28 04:01 libtorch_cuda_cpp.so
    -rwxr-xr-x 1 root root 653M Jul 28 04:01 libtorch_cuda_cu.so
    -rwxr-xr-x 1 root root 209M Jul 28 04:01 libtorch_cuda_linalg.so
    -rwxr-xr-x 1 root root 163K Jul 28 04:01 libtorch_cuda.so
    -rwxr-xr-x 1 root root  21K Jul 28 04:01 libtorch_global_deps.so
    -rwxr-xr-x 1 root root  21M Jul 28 04:01 libtorch_python.so
    -rwxr-xr-x 1 root root  16K Jul 28 04:01 libtorch.so
    
    $ ldd /tmp/tmp.3M50Q7bV1d/venv/lib/python3.7/site-packages/torchopt/_lib/adam_op.cpython-37m-x86_64-linux-gnu.so
            linux-vdso.so.1 =>  (0x00007ffcd44ea000)
            libc10.so => /tmp/tmp.3M50Q7bV1d/venv/lib/python3.7/site-packages/torchopt/_lib/../../torch/lib/libc10.so (0x00007f75c1243000)
            libc10_cuda.so => /tmp/tmp.3M50Q7bV1d/venv/lib/python3.7/site-packages/torchopt/_lib/../../torch/lib/libc10_cuda.so (0x00007f75c109b000)
            libcaffe2_nvrtc.so => /tmp/tmp.3M50Q7bV1d/venv/lib/python3.7/site-packages/torchopt/_lib/../../torch/lib/libcaffe2_nvrtc.so (0x00007f75c123c000)
            libshm.so => /tmp/tmp.3M50Q7bV1d/venv/lib/python3.7/site-packages/torchopt/_lib/../../torch/lib/libshm.so (0x00007f75c1231000)
            libtorch.so => /tmp/tmp.3M50Q7bV1d/venv/lib/python3.7/site-packages/torchopt/_lib/../../torch/lib/libtorch.so (0x00007f75c122c000)
            libtorch_cpu.so => /tmp/tmp.3M50Q7bV1d/venv/lib/python3.7/site-packages/torchopt/_lib/../../torch/lib/libtorch_cpu.so (0x00007f75a704d000)
            libtorch_cuda.so => /tmp/tmp.3M50Q7bV1d/venv/lib/python3.7/site-packages/torchopt/_lib/../../torch/lib/libtorch_cuda.so (0x00007f75c120c000)
            libtorch_cuda_cpp.so => /tmp/tmp.3M50Q7bV1d/venv/lib/python3.7/site-packages/torchopt/_lib/../../torch/lib/libtorch_cuda_cpp.so (0x00007f7599db0000)
            libtorch_cuda_cu.so => /tmp/tmp.3M50Q7bV1d/venv/lib/python3.7/site-packages/torchopt/_lib/../../torch/lib/libtorch_cuda_cu.so (0x00007f757237b000)
            libtorch_cuda_linalg.so => /tmp/tmp.3M50Q7bV1d/venv/lib/python3.7/site-packages/torchopt/_lib/../../torch/lib/libtorch_cuda_linalg.so (0x00007f7565905000)
            libtorch_global_deps.so => /tmp/tmp.3M50Q7bV1d/venv/lib/python3.7/site-packages/torchopt/_lib/../../torch/lib/libtorch_global_deps.so (0x00007f75c1203000)
            libtorch_python.so => /tmp/tmp.3M50Q7bV1d/venv/lib/python3.7/site-packages/torchopt/_lib/../../torch/lib/libtorch_python.so (0x00007f7564957000)
            librt.so.1 => /lib64/librt.so.1 (0x00007f756474f000)
            libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f7564533000)
            libdl.so.2 => /lib64/libdl.so.2 (0x00007f756432f000)
            libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007f7564027000)
            libm.so.6 => /lib64/libm.so.6 (0x00007f7563d25000)
            libgomp.so.1 => /lib64/libgomp.so.1 (0x00007f7563aff000)
            libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f75638e9000)
            libc.so.6 => /lib64/libc.so.6 (0x00007f756351b000)
            /lib64/ld-linux-x86-64.so.2 (0x00007f75c1199000)
            libgomp-a34b3233.so.1 => /tmp/tmp.3M50Q7bV1d/venv/lib/python3.7/site-packages/torchopt/_lib/../../torch/lib/libgomp-a34b3233.so.1 (0x00007f75632f1000)
            libcuda.so.1 => /lib64/libcuda.so.1 (0x00007f7561e96000)
            libnvrtc-4dd39364.so.11.2 => /tmp/tmp.3M50Q7bV1d/venv/lib/python3.7/site-packages/torchopt/_lib/../../torch/lib/libnvrtc-4dd39364.so.11.2 (0x00007f755f075000)
            libcudart-45da57e3.so.11.0 => /tmp/tmp.3M50Q7bV1d/venv/lib/python3.7/site-packages/torchopt/_lib/../../torch/lib/libcudart-45da57e3.so.11.0 (0x00007f755edcd000)
            libnvToolsExt-847d78f2.so.1 => /tmp/tmp.3M50Q7bV1d/venv/lib/python3.7/site-packages/torchopt/_lib/../../torch/lib/libnvToolsExt-847d78f2.so.1 (0x00007f755ebc2000)
            libcudnn.so.8 => /tmp/tmp.3M50Q7bV1d/venv/lib/python3.7/site-packages/torchopt/_lib/../../torch/lib/libcudnn.so.8 (0x00007f755e99a000)
            libcublas.so.11 => /tmp/tmp.3M50Q7bV1d/venv/lib/python3.7/site-packages/torchopt/_lib/../../torch/lib/libcublas.so.11 (0x00007f755521c000)
            libcublasLt.so.11 => /tmp/tmp.3M50Q7bV1d/venv/lib/python3.7/site-packages/torchopt/_lib/../../torch/lib/libcublasLt.so.11 (0x00007f75401b6000)
    
    $ patchelf --print-rpath /tmp/tmp.3M50Q7bV1d/venv/lib/python3.7/site-packages/torchopt/_lib/adam_op.cpython-37m-x86_64-linux-gnu.so
    $ORIGIN/../../torch/lib:$ORIGIN/../../torchopt.libs
    

    Then deploy with torch==1.12.0+cpu:

    $ ls /tmp/tmp.3M50Q7bV1d/venv/lib/python3.7/site-packages/torch/lib
    total 496M
    -rwxr-xr-x 1 root root 269K Jul 28 04:02 libbackend_with_compiler.so
    -rwxr-xr-x 1 root root 766K Jul 28 04:02 libc10.so
    -rwxr-xr-x 1 root root 165K Jul 28 04:02 libgomp-a34b3233.so.1
    -rwxr-xr-x 1 root root 228K Jul 28 04:02 libjitbackend_test.so
    -rwxr-xr-x 1 root root  35K Jul 28 04:02 libshm.so
    -rwxr-xr-x 1 root root 588K Jul 28 04:02 libtorchbind_test.so
    -rwxr-xr-x 1 root root 476M Jul 28 04:02 libtorch_cpu.so
    -rwxr-xr-x 1 root root 8.6K Jul 28 04:02 libtorch_global_deps.so
    -rwxr-xr-x 1 root root  19M Jul 28 04:02 libtorch_python.so
    -rwxr-xr-x 1 root root 7.1K Jul 28 04:02 libtorch.so
    
    $ ldd /tmp/tmp.3M50Q7bV1d/venv/lib/python3.7/site-packages/torchopt/_lib/adam_op.cpython-37m-x86_64-linux-gnu.so
            linux-vdso.so.1 =>  (0x00007ffd5997a000)
            libc10.so => /tmp/tmp.3M50Q7bV1d/venv/lib/python3.7/site-packages/torchopt/_lib/../../torch/lib/libc10.so (0x00007ff16efde000)
            libc10_cuda.so => not found
            libcaffe2_nvrtc.so => not found
            libshm.so => /tmp/tmp.3M50Q7bV1d/venv/lib/python3.7/site-packages/torchopt/_lib/../../torch/lib/libshm.so (0x00007ff16efce000)
            libtorch.so => /tmp/tmp.3M50Q7bV1d/venv/lib/python3.7/site-packages/torchopt/_lib/../../torch/lib/libtorch.so (0x00007ff16efcb000)
            libtorch_cpu.so => /tmp/tmp.3M50Q7bV1d/venv/lib/python3.7/site-packages/torchopt/_lib/../../torch/lib/libtorch_cpu.so (0x00007ff155b9a000)
            libtorch_cuda.so => not found
            libtorch_cuda_cpp.so => not found
            libtorch_cuda_cu.so => not found
            libtorch_cuda_linalg.so => not found
            libtorch_global_deps.so => /tmp/tmp.3M50Q7bV1d/venv/lib/python3.7/site-packages/torchopt/_lib/../../torch/lib/libtorch_global_deps.so (0x00007ff16efc5000)
            libtorch_python.so => /tmp/tmp.3M50Q7bV1d/venv/lib/python3.7/site-packages/torchopt/_lib/../../torch/lib/libtorch_python.so (0x00007ff154dc0000)
            librt.so.1 => /lib64/librt.so.1 (0x00007ff154bb8000)
            libpthread.so.0 => /lib64/libpthread.so.0 (0x00007ff15499c000)
            libdl.so.2 => /lib64/libdl.so.2 (0x00007ff154798000)
            libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007ff154490000)
            libm.so.6 => /lib64/libm.so.6 (0x00007ff15418e000)
            libgomp.so.1 => /lib64/libgomp.so.1 (0x00007ff153f68000)
            libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007ff153d52000)
            libc.so.6 => /lib64/libc.so.6 (0x00007ff153984000)
            /lib64/ld-linux-x86-64.so.2 (0x00007ff16ef39000)
            libgomp-a34b3233.so.1 => /tmp/tmp.3M50Q7bV1d/venv/lib/python3.7/site-packages/torchopt/_lib/../../torch/lib/libgomp-a34b3233.so.1 (0x00007ff15375a000)
    
    $ patchelf --print-rpath /tmp/tmp.3M50Q7bV1d/venv/lib/python3.7/site-packages/torchopt/_lib/adam_op.cpython-37m-x86_64-linux-gnu.so
    $ORIGIN/../../torch/lib:$ORIGIN/../../torchopt.libs
    

    Expected behavior

    A clear and concise description of what you expected to happen.

    Only link libtorch.so and can import shared libraries with different torch dependencies.

    Screenshots

    If applicable, add screenshots to help explain your problem.

    Test torchopt wheels (built with torch==1.12.0+cu116) with torch==1.12.0+cpu.

    image

    System info

    Describe the characteristic of your environment:

    • Describe how the library was installed (pip, source, ...)
    • Python version
    • Versions of any other relevant libraries
    import torchopt, numpy, sys
    print(torchopt.__version__, numpy.__version__, sys.version, sys.platform)
    

    N/A

    Additional context

    Add any other context about the problem here.

    N/A

    Reason and Possible fixes

    If you know or suspect the reason for this bug, paste the code lines and suggest modifications.

    N/A

    Checklist

    • [X] I have checked that there is no similar issue in the repo (required)
    • [X] I have read the documentation (required)
    • [X] I have provided a minimal working example to reproduce the bug (required)
    bug cxx / cuda 
    opened by XuehaiPan 2
  • Got 'nan' when using MetaAdam as a inner_opt

    Got 'nan' when using MetaAdam as a inner_opt

    I just made two modifications on examples/few-shot/maml-omniglot.py:

    1. change the net structure in line 98 to my net:
    net = nn.Sequential(
        nn.Conv2d(1, 64, (3, 3), 2, 0), nn.BatchNorm2d(64), nn.ReLU(),
        nn.Conv2d(64, 64, (3, 3), 2, 0), nn.BatchNorm2d(64), nn.ReLU(),
        nn.Conv2d(64, 64, (3, 3), 2, 0), nn.BatchNorm2d(64), nn.ReLU(),
        nn.Conv2d(64, 64, (2, 2), 1, 0), nn.BatchNorm2d(64), nn.ReLU(),
        nn.Flatten(), nn.Linear(64, args.n_way),
    ).to(device)
    
    1. change the inner_opt from MetaSGD to MetaAdam in line 120.

    Then I got 'nan' parameters after first time of meta_opt.step()

    bug 
    opened by hccz95 2
  • [BUG] This error is raised when using Implicit differentiation module.

    [BUG] This error is raised when using Implicit differentiation module.

    Describe the bug

    The forward is normal, but when I compute gradient, it raised this bug. Seems it does not recognize the parameter that I requires grad, or am I using this function in a wrong way? Please help me in this, Thanks.

    To Reproduce

    def sgf_stationary(params, meta_params, data):
        gs, ee, s00, h00, s01, h01 = params
        left = meta_params[0]
        if not left:
            gs_ = ee*s00 - h00 - (ee * s01 - h01) @ gs @ (ee * s01.conj().T - h01.conj().T)
            gs_ = tLA.pinv(gs_)
        else:
            gs_ = ee*s00 - h00 - (ee * s01.conj().T - h01.conj().T) @ gs @ (ee * s01 - h01)
            gs_ = tLA.pinv(gs_)
    return gs_ - gs
    @torchopt.implicit_diff.custom_root(sgf_stationary, argnums=[1,2,3,4,5], solve=torchopt.linear_solve.solve_normal_cg(maxiter=5, atol=0),)
    def sgf(params, meta_params, data):
        _, ee, S, H, s01, h01 = params
        left, method = meta_params
        if method == 'GEP':
            gs = calcg0(ee, H, S, h01, s01, left=left)
        else:
            h10 = h01.conj().T
            s10 = s01.conj().T
            alpha, beta = h10 - ee * s10, h01 - ee * s01
            eps, epss = H.clone(), H.clone()
            converged = False
            iteration = 0
            while not converged:
                iteration += 1
                oldeps, oldepss = eps.clone(), epss.clone()
                oldalpha, oldbeta = alpha.clone(), beta.clone()
                tmpa = tLA.solve(ee * S - oldeps, oldalpha)
                tmpb = tLA.solve(ee * S - oldeps, oldbeta)
    
                alpha, beta = torch.mm(oldalpha, tmpa), torch.mm(oldbeta, tmpb)
                eps = oldeps + torch.mm(oldalpha, tmpb) + torch.mm(oldbeta, tmpa)
                if left:
                    epss = oldepss + torch.mm(oldalpha, tmpb)
                else:
                    epss = oldepss + torch.mm(oldbeta, tmpa)
                LopezConvTest = torch.max(alpha.abs() + beta.abs())
    
                if LopezConvTest < 1.0e-40:
                    gs = (ee * S - epss).inverse()
    
                    if left:
                        test = ee * S - H - torch.mm(ee * s10 - h10, gs.mm(ee * s01 - h01))
                    else:
                        test = ee * S - H - torch.mm(ee * s01 - h01, gs.mm(ee * s10 - h10))
                    myConvTest = torch.max((test.mm(gs) - torch.eye(H.shape[0], dtype=h01.dtype)).abs())
                    if myConvTest < 1.0e-5:
                        converged = True
                        if myConvTest > 1.0e-8:
                            v = "RIGHT"
                            if left: v = "LEFT"
                            print(
                                "WARNING: Lopez-scheme not-so-well converged for " + v + " electrode at E = %.4f eV:" % ee.real.item(),
                                myConvTest.item())
                    else:
                        print("Lopez-Sancho", myConvTest,
                                "Error: gs iteration {0}".format(iteration))
                        raise ArithmeticError("Criteria not met. Please check output...")
        return gs
    

    The code is in above.

    Traceback (most recent call last):
      File "/data/ADNEGF/calc/surface_green.py", line 370, in <module>
        torch.autograd.grad(gs.sum(), params)
      File "/root/miniconda3/envs/torch/lib/python3.9/site-packages/torch/autograd/__init__.py", line 300, in grad
        return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
    RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn
    

    Expected behavior

    The backward pass can behave as usual.

    import sys, torch, functorch, torchopt
    import torch
    import torch.linalg as tLA
    import scipy.linalg as SLA
    import torchopt
    

    Additional context

    Add any other context about the problem here.

    Reason and Possible fixes

    If you know or suspect the reason for this bug, paste the code lines and suggest modifications.

    Checklist

    • [ ] I have checked that there is no similar issue in the repo. (required)
    • [ ] I have read the documentation. (required)
    • [ ] I have provided a minimal working example to reproduce the bug. (required)
    bug 
    opened by floatingCatty 1
  • [POC] feat(diff/zero_order): add OOP API for zero-order gradient

    [POC] feat(diff/zero_order): add OOP API for zero-order gradient

    Description

    Describe your changes in detail.

    Motivation and Context

    Why is this change required? What problem does it solve? If it fixes an open issue, please link to the issue here. You can use the syntax close #15213 if this solves the issue #15213

    • [ ] I have raised an issue to propose this change (required for new features and bug fixes)

    Types of changes

    What types of changes does your code introduce? Put an x in all the boxes that apply:

    • [ ] Bug fix (non-breaking change which fixes an issue)
    • [X] New feature (non-breaking change which adds core functionality)
    • [ ] Breaking change (fix or feature that would cause existing functionality to change)
    • [ ] Documentation (update in the documentation)
    • [ ] Example (update in the folder of example)

    Checklist

    Go over all the following points, and put an x in all the boxes that apply. If you are unsure about any of these, don't hesitate to ask. We are here to help!

    • [ ] I have read the CONTRIBUTION guide. (required)
    • [ ] My change requires a change to the documentation.
    • [ ] I have updated the tests accordingly. (required for a bug fix or a new feature)
    • [ ] I have updated the documentation accordingly.
    • [ ] I have reformatted the code using make format. (required)
    • [ ] I have checked the code using make lint. (required)
    • [ ] I have ensured make test pass. (required)
    enhancement pytorch feature 
    opened by XuehaiPan 0
  • [Issue Tracker] PyTorch Distributed RPC

    [Issue Tracker] PyTorch Distributed RPC

    This is an issue tracker for the upstream issues:

    • Initialize RPC with large world size:

      • pytorch/pytorch#80017
      • pytorch/pytorch#80104
      • pytorch/pytorch#85607
    • Pass nn.Module and nn.Parameter as RPC argument:

      • pytorch/pytorch#86525
    bug upstream pytorch 
    opened by XuehaiPan 1
  • [Feature Request] Differentiable Adan optimizer support

    [Feature Request] Differentiable Adan optimizer support

    Motivation

    Adan optimizer is a big hit recently, good feature to support for meta learning research

    Resource

    • paper: https://arxiv.org/abs/2208.06677
    • source code: https://github.com/sail-sg/Adan

    Roadmap

    • [ ] low-level alias
    • [ ] high-level Optimizer
    • [ ] high-level MetaOptimizer
    • [ ] low-level test
    • [ ] high-level Optimizer test
    • [ ] api docs

    Checklist

    • [x] I have checked that there is no similar issue in the repo (required)
    enhancement 
    opened by Benjamin-eecs 0
Releases(v0.6.0)
  • v0.6.0(Dec 6, 2022)

    Highlights

    • The implicit gradient and zero-order differentiation APIs become generally available in TorchOpt 0.6.0.
    • The project code structure is refactored and reorganized.

    [0.6.0] - 2022-12-07

    Added

    • Add unroll pragma for CUDA OPs by @JieRen98 and @XuehaiPan in #112.
    • Add Python implementation of accelerated OP and pure-Python wheels by @XuehaiPan in #67.
    • Add nan_to_num hook and gradient transformation by @XuehaiPan in #119.
    • Add matrix inversion linear solver with neumann series approximation by @Benjamin-eecs and @XuehaiPan in #98.
    • Add if condition of number of threads for CPU OPs by @JieRen98 in #105.
    • Add implicit MAML omniglot few-shot classification example with OOP APIs by @XuehaiPan in #107.
    • Add implicit MAML omniglot few-shot classification example by @Benjamin-eecs in #48.
    • Add object-oriented modules support for implicit meta-gradient by @XuehaiPan in #101.
    • Bump PyTorch version to 1.13.0 by @XuehaiPan in #104.
    • Add zero-order gradient estimation by @JieRen98 in #93.
    • Add RPC-based distributed training support and add distributed MAML example by @XuehaiPan in #83.
    • Add full type hints by @XuehaiPan in #92.
    • Add API documentation and tutorial for implicit gradients by @Benjamin-eecs and @JieRen98 and @XuehaiPan in #73.
    • Add wrapper class for functional optimizers and examples of functorch integration by @vmoens and @Benjamin-eecs and @XuehaiPan in #6.
    • Implicit differentiation support by @JieRen98 and @waterhorse1 and @XuehaiPan in #41.

    Changed

    • Refactor code organization by @XuehaiPan in #92 and #100.

    Fixed

    • Fix implicit MAML omniglot few-shot classification example by @XuehaiPan in #108.
    • Align results of distributed examples by @XuehaiPan in #95.
    • Fix None in module containers by @XuehaiPan.
    • Fix backward errors when using inplace sqrt_ and add_ by @Benjamin-eecs and @JieRen98 and @XuehaiPan.
    • Fix LR scheduling by @XuehaiPan in #76.
    • Fix the step count tensor (shape=(1,)) can change the shape of the scalar updates (shape=()) by @XuehaiPan in #71.

    Full Changelog: https://github.com/metaopt/torchopt/compare/v0.5.0...v0.6.0

    Source code(tar.gz)
    Source code(zip)
  • v0.5.0(Sep 5, 2022)

    [0.5.0] - 2022-09-05

    Added

    • Implement AdamW optimizer with masking by @Benjamin-eecs and @XuehaiPan in #44.
    • Add half float support for accelerated OPs by @XuehaiPan in #67.
    • Add MAML example with TorchRL integration by @vmoens and @Benjamin-eecs in #12.
    • Add optional argument params to update function in gradient transformations by @XuehaiPan in #65.
    • Add option weight_decay option to optimizers by @XuehaiPan in #65.
    • Add option maximize option to optimizers by @XuehaiPan in #64.
    • Refactor tests using pytest.mark.parametrize and enabling parallel testing by @XuehaiPan and @Benjamin-eecs in #55.
    • Add maml-omniglot few-shot classification example using functorch.vmap by @Benjamin-eecs in #39.
    • Add parallel training on one GPU using functorch.vmap example by @Benjamin-eecs in #32.
    • Add question/help/support issue template by @Benjamin-eecs in #43.

    Changed

    • Align argument names with PyTorch by @XuehaiPan in #65.
    • Replace JAX PyTrees with OpTree by @XuehaiPan in #62.
    • Update image link in README to support PyPI rendering by @Benjamin-eecs in #56.

    Fixed

    • Fix RMSProp optimizer by @XuehaiPan in #55.
    • Fix momentum tracing by @XuehaiPan in #58.
    • Fix CUDA build for accelerated OP by @XuehaiPan in #53.
    • Fix gamma error in MAML-RL implementation by @Benjamin-eecs #47.
    Source code(tar.gz)
    Source code(zip)
    torchopt-0.5.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl(490.23 KB)
    torchopt-0.5.0-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl(489.10 KB)
    torchopt-0.5.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl(490.19 KB)
    torchopt-0.5.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl(490.50 KB)
    torchopt-0.5.0.tar.gz(50.39 KB)
  • v0.4.3(Aug 7, 2022)

    [0.4.3] - 2022-08-08

    What's Changed

    Added

    • Bump PyTorch version to 1.12.1 by @XuehaiPan in #49.
    • CPU-only build without nvcc requirement by @XuehaiPan in #51.
    • Use cibuildwheel to build wheels by @XuehaiPan in #45.
    • Use dynamic process number in CPU kernels by @JieRen98 in #42.

    Changed

    • Use correct Python Ctype for pybind11 function prototype @XuehaiPan in #52.

    Full Changelog: https://github.com/metaopt/TorchOpt/compare/v0.4.2...v0.4.3 Pre-built wheels: https://pypi.org/project/torchopt/0.4.3/#files

    Source code(tar.gz)
    Source code(zip)
    torchopt-0.4.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl(348.04 KB)
    torchopt-0.4.3-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl(347.16 KB)
    torchopt-0.4.3-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl(347.97 KB)
    torchopt-0.4.3-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl(348.20 KB)
    torchopt-0.4.3.tar.gz(43.89 KB)
  • v0.4.2(Jul 26, 2022)

    [0.4.2] - 2022-07-26

    What's Changed

    Added

    • Read the Docs integration by @Benjamin-eecs and @XuehaiPan in #34.
    • Update documentation and code styles by @Benjamin-eecs and @XuehaiPan in #22.
    • Update tutorial notebooks by @XuehaiPan in #27.
    • Bump PyTorch version to 1.12 by @XuehaiPan in #25.
    • Support custom Python executable path in CMakeLists.txt by @XuehaiPan in #18.
    • Add citation information by @waterhorse1 in #14 and @Benjamin-eecs in #15.
    • Implement RMSProp optimizer by @future-xy in #8.

    Changed

    • Use pyproject.toml for packaging and update GitHub Action workflows by @XuehaiPan in #31.
    • Rename the package from TorchOpt to torchopt by @XuehaiPan in #20.

    Fixed

    • Fixed errors while building from the source and add conda environment recipe by @XuehaiPan in #24.

    Full Changelog: https://github.com/metaopt/TorchOpt/compare/v0.4.1...v0.4.2

    Source code(tar.gz)
    Source code(zip)
  • v0.4.1(Jun 14, 2022)

Owner
MetaOPT Team
MetaOPT Team
Uni-Core is built for rapidly creating PyTorch models with high performance, especially for Transfromer-based models

Uni-Core, an efficient distributed PyTorch framework Uni-Core is built for rapidly creating PyTorch models with high performance, especially for Trans

DP Technology 37 Dec 16, 2022
Pytoydl: A toy deep learning framework built upon numpy.

Documents: https://pytoydl.readthedocs.io/zh/latest/ Pytoydl A toy deep learning framework built upon numpy. You can star this repository to keep trac

null 28 Dec 10, 2022
Betty: an automatic differentiation library for generalized meta-learning and multilevel optimization

An automatic differentiation library for generalized meta-learning and multilevel optimization Docs | Tutorials | Examples | Paper | CASL Project pip

leopard-ai 213 Dec 17, 2022
This repository contains the source code of a fully functional "Note Pad" Application built in Python language using PyQt5 library.

Note Pad Application in Python This repository contains the source code of a fully functional "Note Pad" Application built in Python language using

null 1 Oct 9, 2022
Gradient-based constrained optimization for JAX

mdmm-jax Gradient-based constrained optimization for JAX (implementation by Katherine Crowson). The Modified Differential Multiplier Method was propos

Katherine Crowson 11 Dec 16, 2022
Contribution to an open source repository which implements the Bayesian Optimization algorithm - Knowledge Gradient implementation

Authors: Raffaella D'Anna Michele Di Sabato Martina Garavaglia Anna Iob Veronica Mazzola Open source project: original repository: https://github.com/

Michele Di Sabato 2 Nov 27, 2022
An OCR based monitor for UK smart meters based upon the Raspberry Pi that provides metrics via a Prometheus endpoint.

Smart Meter OCR Monitor An OCR based monitor for UK smart meters based upon the Raspberry Pi that provides metrics via a Prometheus endpoint. Licence

Sidings Media 1 Jun 12, 2022
Implementation of the Adan (ADAptive Nesterov momentum algorithm) Optimizer in Pytorch

Adan - Pytorch Implementation of the Adan (ADAptive Nesterov momentum algorithm) Optimizer in Pytorch. Explanation from Davis Blalock Official Adan co

Phil Wang 215 Dec 24, 2022
Official code for ''From Optimization Dynamics to Generalization Bounds via Łojasiewicz Gradient Inequality'' (TMLR)

From Optimization Dynamics to Generalization Bounds via Łojasiewicz Gradient Inequality This repository contains the official codes that used for gene

mathlfs 3 Oct 30, 2022
FairGrad, is an easy to use general purpose approach to enforce fairness for gradient descent based methods.

FairGrad: Fairness Aware Gradient Descent FairGrad, is an easy to use general purpose approach to enforce fairness for gradient descent based methods.

Gaurav Maheshwari 10 Dec 21, 2022
Service Unit tracking for SLURM, python3 - based upon slurm_bank

slurm_su_bank_python3 UNDER DEVELOPMENT!!! USE AT OWN RISK A Banking/Resource Allocation (Service Unit) tracking system for the SLURM job scheduler ba

David Simpson 1 Apr 8, 2022
A functional, easy to use Python wrapper of Notion Api.

Potion A functional, easy to use Python wrapper of Notion Api. easy-to-use low-level api integration local-backup function render notion object to mar

sail 16 Nov 24, 2022
A Python high-performance screen capture library for Windows using Desktop Duplication API

DXcam Fastest Python Screenshot for Windows import dxcam camera = dxcam.create() camera.grab() Introduction DXcam is a Python high-performance screens

Rain 132 Jan 8, 2023
Pond is a high performance object-pooling library for Python

Pond Pond is a high performance object-pooling library for Python, it has a smaller memory footprint and a higher hit rate. For more details, see our

Andy 32 Dec 30, 2022
A high-performance inter-process communication library designed to handle communication between multiple shards

Better Cluster A high-performance inter-process communication library designed to handle communication between multiple shards This library is made to

DaPandaOfficial 2 Aug 28, 2022
[ECCV 2022] PillarNet: Real-Time and High-Performance Pillar-based 3D Object Detection

PillarNet: Real-Time and High-Performance Pillar-based 3D Object Detection Pillar-based single-stage 3D object detector using center-based head in the

SJTU Vision and Learning Group 10 Jan 3, 2023
Official implementation of "Gradient-Guided Importance Sampling for Learning Binary Energy-Based Models" [Preprint]

Gradient-Guided Importance Sampling for Learning Binary Energy-Based Models This is the official implementation of the RMwGGIS method proposed in the

DIVE Lab, Texas A&M University 6 Dec 4, 2022
FastAPI is a modern, fast (high-performance), web framework for building APIs with Python 3.6+ based on standard Python type hints.

fastapi_ FastAPI is a modern, fast (high-performance), web framework for building APIs with Python 3.6+ based on standard Python type hints. Em portug

Marcus 2 Jun 10, 2022